Method and system for automated quality assurance and automated treatment planning in radiation therapy

ABSTRACT

Methods and systems for evaluating a proposed treatment plan for radiation therapy, for evaluating one or more delineated regions of interest for radiation therapy, and/or for generating a proposed treatment plan for radiation therapy. Machine learning based on historical data may be used.

TECHNICAL FIELD

The present disclosure relates to methods and systems for evaluating andgenerating radiation therapy (RT) treatment plans. In particular, thepresent disclosure relates to methods and systems for providing acalculated quality estimate for a proposed RT treatment plan and/or fora delineated region of interest and generating a proposed dosedistribution for an RT treatment plan.

BACKGROUND

An estimated 187,600 new cases of cancer are expected in Canada in 2013[1] with radiation therapy (RT) indicated as part of the patient'smanagement in approximately 40 percent of cancer cases [2]. The deliveryof RT for the treatment of cancer typically is a complicated processthat requires both clinical and technical expertise in order to generatetreatment plans that are safe and effective for the treatment of cancer.

For the RT process, patients are imaged with computed tomography (CT)imaging and optionally with multi-modality imaging (e.g. MR, PET)depending on the treatment site. Regions of interest (ROIs) i.e. targets(the locations radiation is directed to) and normal tissue structures(the locations radiation is minimized to) are delineated manually and/orsemi-automatically on the acquired images (a). Treatment plans aregenerated manually, in which the direction of radiation beams and theclinical objectives of the treatment must be specified. An optimizationalgorithm is then used to generate the intensity and/or shape and/ormodulation of radiation beams to achieve the treatment objectives (b). Adose distribution, a spatial representation of the radiation dose thepatient will receive, can then be calculated. Therefore, the dosedistribution (also referred to as a dose map) is directly connected withthe anatomical imaging acquired from the RT process to relate the doseand spatial information specific to the patient.

In addition, the dose distribution is used to quantitatively evaluatethe dose received by the delineated ROIs for assessing treatment planquality and safety (c). The steps (a-c) are repeated until an acceptableplan is generated. Finally, the completed treatment plans are thenreviewed by the multi-disciplinary RT team for quality, safety andcompliance with established clinical protocols before the treatment planwill be delivered to the patient.

RT Quality Assurance

The RT treatment plan quality assurance (QA) process typically relies onthe vigilance of the multi-disciplinary team to review and assimilaterelatively complex data from different sources. Human vigilance has beenfound to be effective in the treatment plan QA process in about 80percent of cases [3] and for preventing treatment incidents in about 98percent of cases [4,5]. As a result, sub-optimal treatment plans, whichhave the potential to result in a significant detriment to the patient,may be used clinically. Several studies have shown treatment plans,which deviate from established QA guidelines, result in worse patientoutcomes [6,7]. Therefore, the current RT process may requiresubstantial multi-disciplinary QA resources to reduce the likelihood oferrors and to ensure a high standard of patient care.

The multi-disciplinary RT team comprising radiation therapists,physicists and oncologists typically reviews each proposed treatmentplan for clinical and technical merit. This review typically includesassessing safety (e.g., that the proposed plan does not exceed anynormal tissue dose tolerances), deliverability (e.g., the dosecalculated in the proposed treatment plan can be reproduced on thetreatment unit), consistency in the transfer of data between databases(e.g., the parameters defining the proposed plan are the same parametersto actually treat the specific patient) and overall quality (e.g., theproposed plan is consistent with other plans for the given site andtechnique in terms of the dose prescription, the dose distribution,target coverage etc.) [8-19].

This process is typically largely manual and complex, as there may benumerous parameters that require human expert review. This has lead toan interest in automated QA methods in order to reduce the reliance onhuman vigilance [20-23]. Methods developed to date have shown promiseonly in a limited clinical scope.

RT Planning

Technical innovations in RT have improved the quality of treatment plansusually at the cost of increased complexity. However, treatment planningstill remains a highly manual process, which requires users to delineatenumerous regions of interest (ROIs) for treatment planning and settreatment objectives for an optimization engine to solve. For example,optimization objectives may specify the target ROI must receive >95% ofthe prescription dose to >95% of the target volume while a healthy organmust receive <100% of the prescription dose to 1 cc of the organ volume.

The process almost always involves multiple iterations, as changes tothe objectives and the ROIs themselves are required to generate anacceptable treatment plan. To date, conventional automated treatmentplanning methods have focused on setting objectives and then optimizingthose objectives to generate the dose distribution (also referred to asa dose map). Such a process still requires ROI delineation, beamplacement, and manual adjustment of the objectives.

In addition, the variation in ROI delineation and treatment plan qualityis well-established [24, 25]. The use of automation may help to improveconsistency and add standardization to the process [26].

SUMMARY

In various examples and embodiments, the present disclosure incorporatesautomation into the multi-disciplinary RT treatment plan QA andtreatment planning process. This may help to improve plan quality and/orpatient safety above the 80 percent level attained by human vigilancealone.

In some examples, automation in the context of QA and treatment planningmay employ computer-assisted methods such as machine learning (e.g.,classification and regression) and/or registration (e.g., imageprocessing). The present disclosure provides examples employing themachine learning techniques of automated classification and regression;however other computer-assisted techniques could be used to achieve anautomated QA and/or treatment planning process.

An automated QA and/or treatment planning framework, such as thatprovided in some examples of the present disclosure, may help to reducedelays in patients receiving treatment due to errors necessitating planre-work and re-optimization. Automated classification and/or regressionalgorithms may better utilize the vast clinical RT data available and/ormay provide a mechanism for standardizing and/or continuously improvingthe quality of plans by correlating individual treatment planning withplans of known high quality and safety. The methodology may promotewidespread dissemination to other institutions, which may benefit otherpatients receiving RT as part of their cancer management.

Compared to conventional simple automated QA and treatment planningmethods, the present disclosure may employ more sophisticated algorithmsand more data that may have the potential to have clinical impact acrossa much wider array of treatments.

In some examples, the present disclosure describes using a database ofpatients with corresponding high-quality treatment plans, which may helpto enable one or more of: automatically inferring the quality of a novelproposed plan for a novel patient; automatically inferring the qualityof delineated regions of interest (ROIs); automatically inferring theclass label for each ROI; and automatically inferring a class label forthe proposed treatment plan, estimating a dose distribution specific toa novel patient, inferring the treatment planning parameters to achievethe estimated dose distribution.

A basic mechanism of this approach may be to figure out which patientsin a database of historical treatment plans and patients are mostsimilar to the novel patient, and then 1) estimating a dose distributionand associated treatment plan or 2) compare the novel proposed plan withthe corresponding historical patient plans from the database. Though itmay be possible to manually encode distance metrics between patients,and/or distance metrics between plans, in order to evaluate similarity,in some examples machine learning methods may be used to automaticallylearn the relationship between different patients and/or differentplans, and ultimately patients and plans.

In some example aspects, the present disclosure provides a method forevaluating a proposed treatment plan for RT, where the method mayinclude: obtaining the proposed treatment plan defining treatment for atleast one treatment site, and a set of patient data for a patient;automatically characterizing the proposed treatment plan according toone or more predefined features to determine a treatment plancharacterization; calculating a quality estimate for the proposedtreatment plan by evaluating the proposed treatment plan according toone or more rules (which may be predefined or learned by the system, forexample, as discussed further below) defining expected relationshipsbetween the treatment plan characterization and one or more of: one ormore plan features, and one or more patient features defined in the setof patient data; and providing output indicating the calculated qualityestimate.

In some examples, the method may also include: obtaining a set of regionof interest (ROI) data delineating at least one ROI in the set of imagedata; and automatically characterizing the at least one ROI according toone or more predefined features to determine at least one ROIcharacterization; wherein calculating the quality estimate for theproposed treatment plan includes evaluating the proposed treatment planaccording to one or more rules defining expected relationships betweenthe treatment plan characterization and the at least one ROIcharacterization.

In some example aspects, the present disclosure provides a method forevaluating at least one delineated region of interest (ROI) for RT,where the method may include: obtaining a set of ROI data delineatingthe at least one ROI for at least one treatment site in a set of imagedata, and a set of patient data for a patient; automaticallycharacterizing the at least one ROI according to one or more predefinedfeatures to determine at least one ROI characterization; calculating aquality estimate for the at least one ROI by evaluating the at least oneROI according to one or more rules (which may be predefined or learnedby the system, for example, as discussed further below) definingexpected relationships between the at least one ROI characterization andone or more of: one or more ROI features, and one or more patientfeatures defined in the set of patient data; and providing outputindicating the calculated quality estimate.

In some examples, the method may include: obtaining a proposed treatmentplan defining treatment for the at least one treatment site; andautomatically characterizing the proposed treatment plan according toone or more predefined features to determine a treatment plancharacterization; wherein calculating the quality estimate for the atleast one ROI includes evaluating the at least one ROI according to oneor more rules defining expected relationships between the at least oneROI characterization and the treatment plan characterization.

In some example aspects, the present disclosure provides a method forgenerating a proposed treatment plan for radiation therapy, where themethod may include: obtaining a set of patient data for a patientincluding at least one set of image data for at least one treatmentsite; determining a treatment plan class, from a plurality of predefinedtreatment plan classes, each predefined treatment plan classes definingone or more treatment plan features relevant to treatment of arespective treatment site; calculating a proposed dose map (e.g., byperforming dose inference) by determining a dosage over a volumedepicted in the set of image data according to a first set of rules(which may include predefined rules or machine-learned rules, asdiscussed further below) including one or more rules defining expectedrelationships between applied dosage, the treatment plan class, and atleast one feature of the image data; determining one or more treatmentplan parameters for achieving the proposed dose map according to asecond set of rules (which may include predefined rules ormachine-learned rules, as discussed further below) defining expectedrelationships between dosage and treatment plans; and generating asoutput the proposed treatment plan including the one or more determinedtreatment plan parameters.

In some examples, the method may include displaying a visualization ofthe proposed dose map on a display device, the visualization comprisinga voxel-by-voxel mapping of proposed dosages superimposed on the set ofimage data. Optionally, a user interface may be provided with thevisualization to receive user input to modify at least one of: proposeddosage for at least one voxel of the proposed dose map, and one or morefeatures of the image data. The proposed dose map may then berecalculated in accordance with any inputted modification.

In some example aspects, the present disclosure provides a system forevaluating a proposed treatment plan for RT, for evaluating at least oneROI for RT and/or for generating a proposed treatment plan for RT, wherethe system may include a processor configured to executecomputer-readable instructions that, when executed, causes the system tocarry out any of the above methods.

In some examples, the system may include a web client for providing theoutput (e.g., via a web-based portal). The system may include one ormore databases for storing one or more rules and/or historical data, asdescribed further below.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the drawings, which show exampleembodiments of the present disclosure, and in which:

FIG. 1 is a flowchart showing an overview of an example method forautomated quality assurance;

FIG. 2 is a block diagram of an example system for carrying examples ofthe disclosed methods;

FIGS. 3a and 3b show a table of example ROI features;

FIGS. 4a-4d are diagrams illustrating progressively descriptive graphstructures for examples of automatic ROI classification;

FIG. 5 is a table of notation used in the present disclosure;

FIG. 6 is a table of group wise features calculated between ROIs;

FIGS. 7a-7b are tables of example results comparing different machinelearning algorithms that may be suitable for carrying out classificationof ROIs and treatment plans;

FIGS. 8a-8d are charts illustrating parameter sensitivity in exampleRandom Forest and conditional Random Forest models;

FIGS. 9a-9b are charts illustrating feature importance learned inexample Random Forest and conditional Random Forest models;

FIG. 10 is a chart illustrating confidence interval for detection ofcontour drawing errors in ROIs, in an example of the present disclosure;

FIGS. 11a-11c illustrate an example of a false positive in a delineationof lung ROI;

FIGS. 12a-12c illustrate an example of a false negative in a delineationof heart ROI;

FIGS. 13a-13c show a table showing example results of classification oftreatment plans;

FIGS. 14a-14c and 15a-15c show tables showing example results ofclassification of ROIs;

FIGS. 16a-16b show charts illustrating confidence interval for detectionof errors in treatment plans, in an example of the present disclosure;

FIGS. 17a-17f show a table of example clinically relevant features thatmay be considered for evaluation of treatment plans;

FIG. 18 is a flowchart of an example method for evaluating a proposedtreatment plan;

FIG. 19 is a flowchart of an example method for evaluating a ROI;

FIG. 20a is a flowchart showing an overview of an example method forautomated treatment planning;

FIG. 20b is a more detailed flowchart of an example method for automatedtreatment planning;

FIG. 21 shows a table of example features that may be used forgenerating a proposed dose map (e.g., by performing dose inference);

FIGS. 22a-22c illustrate an example of generating a proposed dose mapfor automated treatment planning; and

FIG. 23 shows a table showing example results of a generated proposeddose map for automated treatment planning.

DETAILED DESCRIPTION

Reference will now be made to the accompanying drawings, which showexample embodiments of the present disclosure. For simplicity andclarity of illustration, reference numerals may be repeated among thefigures to indicate corresponding or analogous elements. Numerousdetails are set forth to provide an understanding of the exampleembodiments described herein. The example embodiments may be practisedwithout some of these details. In other instances, suitable variationsto the disclosed methods, procedures, and components have not beendescribed in detail to avoid obscuring the example embodimentsdescribed, but are within the scope of the present disclosure. Thedescription is not to be considered as limited to the scope of theexample embodiments described herein.

Conventional methods of QA and treatment planning developed to date haveshown promise in a limited clinical scope, however the use of moresophisticated algorithms and more data, as in various examples of thepresent disclosure, may have the potential to have clinical impactacross a much wider array of treatments. This may include applicabilityacross most or all treatment sites and treatment techniques.

To date there have been attempts to do simple classification oftreatment plans for the purposes of establishing quality. Thesetypically have been limited to single treatment plan classes, treatmentsites and/or treatment techniques [19-23]. A possible challenge withconventional QA systems is that they typically do not have any learningcomponent but rather compare data in treatment plans with hard-codedlimits and data. Treatment site-specific templates can also be employed,however conventional templates typically cannot incorporate suitabledeviations without accurate distributions of acceptable treatmentplanning data. As well, tools have been used to check for consistencybetween multiple sources of data, such as between the treatment planningsystem and the databases that contain data to treat the patient at thetreatment unit (e.g., RT electronic medical record system and/oroncology information system).

Other conventional systems have been designed to automatically capturedose information from a novel proposed plan and compared the data withan established protocol template. Again, these conventional systemstypically do not have any learning or classification components.

In some examples, the disclosed methods and systems include learning,classification, regression, and quality estimation components for ROIsand/or quality estimation that jointly consider patient and/or planand/or ROI features. While the details vary between the key applications(ROI quality estimation, plan quality estimation), each method is anovel application for automated learning in the RT context.

Examples of the disclosed methods and systems can also improvethemselves over time and repeatedly, for example using a framework forrepeated training and learning. In some examples, the disclosed methodsand systems may not be limited to a single site or technique but maygeneralize to multiple or all possible treatment sites and/ortechniques, for example via the machine learning component withsufficient training data. This may be different from conventionalapproaches that are typically limited to single treatment sites andtypically must be manually adapted to other sites, for example due tothe lack of machine learning and/or generalized patient and planfeatures.

In some examples, the present disclosure may be extendable to includeuse of potentially multiple machine learning algorithms to learndifferent components of plans. For example, the present disclosurediscusses automated QA for treatment plans and ROIs and automatedtreatment planning and dose inference, but the system may also considerother facets of patient care such as beams or other image features,among other possibilities. Characterization (e.g., classification) ofdifferent aspects (e.g., ROI, images, patient data and/or treatmentplan) of RT treatment may be considered together in order to evaluateeach based on consideration of each other aspect. This may provide auseful coarse-to-fine approach.

In some examples, the disclosed methods and systems may include learningfeatures of the images, the treatment plans and also the ROIs, which aredefined in the treatment plans, depending on what data is available in agiven application. Dose inference, for example, may include zero or moreROIs to calculate features and may include zero or more previous plansfor the patient or an estimate of the plan from a different source (e.g.different system, or user), but the final plan is not available.Treatment plan QA may also include a set of ROIs. This may provide moreinformation and may also provide the ability to use ROI information thatmay enhance the features that can be applied. This may also allowdynamic multi-parameter evaluation incorporating both confidence levels.New data may automatically be used for future learning and testing insome examples. Therefore examples of the present disclosure may provideevaluation and also be adding to the database used for learning, whichmay result in a quality improvement cycle and may enable perpetuallearning, to make the system more robust.

Conventional methods are typically limited in the factors that areconsidered when performing QA or treatment planning. The presentdisclosure may enable such factors to be taken into consideration. Forexample, in the case of dose prediction, this may include patienttreatment history and genetic biomarkers that can be used as additionalfeatures. For automated treatment plan and/or ROI QA these may include,for example, delineated ROIs throughout an entire image(s) as opposed tothe subset that is reviewed, the position of each leaf used to definetreatment beams and segments in intensity modulated radiation therapy(IMRT) and volumetric modulated arc therapy (VMAT), and/or the jointassociation of the treatment plan features relative to ROI volume,location and shape, among others.

A classification and learning system may be able to incorporate one ormore of these and other factors implicitly, without requiring a user todefine specific parameters.

There have been conventional attempts to develop automated treatmentplanning based on limited anatomical and treatment plan data and appliedto a single or particular treatment site/technique [27-34]. Conventionalautomated treatment planning methods using inverse planning approaches(e.g., IMRT and VMAT), often attempt to conform to the standardtreatment planning paradigm by inferring optimization objectives basedon the delineated ROIs. A possible challenge with these methods is thatthey always require the use of ROIs to facilitate the process and oftenrequire specific planning ROIs to be created in order to generate anacceptable treatment plan.

Optimization objectives are the clinical goals for an individual ROIspecified by the user. Optimization objectives can be specified, forexample, as the maximum, minimum or mean dose to a specific volume of anROI. An optimization may specify the target ROI must receive >95% of theprescribed dose to >95% of the target volume. An optimization objectivefor a healthy organ ROI may specific the ROI must receive <100% of theprescription dose to 1 cc of the ROI volume, where the particular doseand volume have an associated complication or toxicity at that level.

The concept of optimization objectives for ROIs was introduced in thefirst implementation of IMRT, in which clinical goals were assigned forROIs to drive the optimization process [35]. The use of ROIs allows thetreatment planner (dosimetrist) the ability to articulate specificobjectives to delineated structures in the plan. In addition, thisenables the process to be iterated by adjusting existing objectives,adding objectives or deleting objectives. The optimization engines usedto design treatment plans were predicated on this approach of specifyingoptimization objectives, which could readily be solved (althoughtypically not optimally).

In the present disclosure, there is no absolute requirement on ROIs tobe delineated to facilitate the automated treatment planning process.ROIs, if included, can be used to drive the learning and inference forautomated treatment planning.

In addition, the optimization objectives used in conventional approacheswere introduced by the clinical RT community, as optimization objectivessuch as these simplify the plan evaluation process, are related torelevant clinical goals and enable direct correlation with knownclinical endpoints. However these optimization objectives are typicallydifficult to optimize, as objectives specified by dose and volume arepercentiles of the dose distribution, which are non-convex from anoptimization perspective. Optimization methods are better for dealingwith mean and maximum doses and quadratic penalties.

In the present disclosure, the concept of optimization objectives arenot required, as dose is not specified to the ROIs but rather the entiredose distribution is inferred. Therefore, instead of specifyingoptimization objectives for which there is a loss of spatialinformation, the present disclosure enable inferring the dose to eachregion of the patient image and encoding spatial information at the sametime. Such an approach has not been implemented conventionally, possiblybecause it involves several components that are together complex. First,there is use of a database of previous treatment plans and imaging forthe same treatment technique that can be trained and tested. Next,algorithms are used to estimate the dose distribution for a novelpatient based on the previous treatment plans in the database. Finally,optimization methods are used to optimize the beam fluence required toreproduce the estimated dose distribution [36, 37]. The beam fluence isthe two-dimensional representation of the RT beam relating the spatiallocation in the beam to the beam intensity. Therefore the fluencedefines the intensity modulation of a given beam, which is characterizedby the shape and/or intensity of the RT beam(s) and/or the shape and/orintensity of the control point(s) for each RT beam. The optimizationmethods for optimizing the beam fluence in the present disclosure maypresent an easier problem for optimization algorithms, as there is nodependence on dose-volume objectives to be solved explicitly.

Example System Overview

FIG. 2 shows an overview of an example system suitable for carrying outexamples of the present disclosure, as will be discussed further below.The example system may include any suitable computing devices including,for example, one or more servers and/or one or more mobile computingdevices (e.g., handheld devices, tablets, mobile communication devices,laptops and smartphones). The example system may communicate with othersystems (e.g., external databases or external servers). Although FIG. 2illustrates scanners 201, treatment planning system 202, RT electronicmedical record system 209 and/or oncology information system and clients203 in addition to an automated QA and planning system, the examplesystem may include only the automated QA and planning system, with theother components being separate from and external to the system.

The example system may include a processor that communicates with one ormore memories storing data and computer-readable instructions that, whenexecuted by the processor, causes the system to carry out the disclosedmethods.

The system may include one or more databases 206 storing the one or morerules and/or historical data. The system may also communicate with oneor more external databases (not shown) for accessing such information,for example. Information in the database(s) may be updated (e.g., as newpatient data and new treatment plans are available).

The example of FIG. 2 may implement a web-based module and visualizationengine. The modular design of the system may limit the scope of eachmodule to a specific task. This may allow a module to be reused in othersystems. For example, the Artificial Intelligence (AI) module 205 may beused by a web application 204 for plan quality review and/or forautomated treatment planning. The same module can also be used by atreatment planning system (discussed further below) to aid in creatingROIs, for example.

Modular design of the example system may also allow it to scale tosupport a growing user base. Each piece can run on separate clusters ofservers. Alternatively, to save on costs it can all run on the sameserver.

This modular design may enable separate presentation of various views ofthe data—which may be running on a client 203—from the massive amount ofplan related data—which may be stored on a separate server. Money can beinvested in high quality server hardware capable of storing andprocessing large volumes of data, while the client equipment can berelatively basic (such as a laptop or a tablet), to review portions ofthe data served to the client over the network.

In some examples, the system may include a web application 204 forreceiving user input and providing the output, where the output isweb-accessible. This web 2.0 application 204 may tie together variousalgorithms and technologies and may present them to the user via a webinterface. The web application 204 may be an implementation of aworkflow engine for a radiation treatment plan—e.g., data constituting aproposed treatment plan may be passed through a rigorous clinicalapproval process with artificial intelligence algorithms aiding eachstep of the process, and may require a human user to sign off. Throughthe web application 204, the user can see the automatic artificialintelligence algorithm quality assessment, visualize all the RT Planelements including patient scans, delineated regions of interest, beamsand computed dose, and finally mark the plan as approved or rejected fordelivery, for example.

The web application may be stand-alone, or link to existing QA orplanning software suites, for example.

The web application 204 may allow ubiquitous access. Users can usedesktops, laptops and tablets, or any other suitable computing device toaccess the service. Users may have the ability to access the application204 from anywhere in the world (e.g., using an internet connection).

All traffic may be encrypted (e.g., via HTTPS) and users may required tobe authenticated, to ensure security and patient privacy.

The example system may include an artificial intelligence (AI) module205. This module 205 may implement machine learning classification andquality assurance algorithms, such as described below. The module 205may expose its functionality via a remote procedure that calls API.

The example system may include a render server DICOM RT visualizationmodule 207. The DICOM RT Visualization module 207 may handle renderrequests for DICOM treatment plan data. Algorithms in this module 207may be able to render various scanned images, beams, regions ofinterest, and dose. The module 207 may be also able to computedose-volume histograms. The module 207 may expose its functionality viaa remote procedure that calls API. This may allow web-based thin clients(such as the web application 204 running in a web browser) to renderDICOM RT data for the user and may let the user to interactivelyvisualize the data.

The example system may include one or more rules and data databases 206,which may store historical data (e.g., plan features and patientfeatures) and rules for implementing automated QA and/or automatedtreatment planning, as discussed below. The machine learning algorithmsmay be trained on a relatively large set of features. This database(s)206 may be a storage location for all those features. The database(s)206 may include a collection of high quality training data extractedfrom tens of thousands of clinical plans.

The example system may include a DICOM database 208. This may be arepository for storing DICOM treatment plan data. A high performance,DICOM compliant storage system may need to be used here to handle therelatively large number (e.g., tens of thousands) of plans the systemmay be expected to handle.

Automated QA: Overview of Example Method and System

FIG. 1 shows an overview of an example method for quality assurance ingreater detail, as will be discussed further below.

In some examples, the disclosed methods and systems may use machinelearning to build an automated method for treatment plan QA. Imaging andtreatment planning data may be grouped according to treatment site andtreatment technique. Clinically relevant features (e.g., as shown inTable 8, FIGS. 17a-17f ) may be selected by a multi-disciplinary RT teamto ensure that the automated learning is based on diverse knowledge.

Features may be extracted from historical training data, and used toconstruct the learning algorithms and train the QA model. Additionalfeatures, not found directly in the treatment plan, may be automaticallygenerated by applying segmentation algorithms which can delineaterelevant organs in the RT image dataset.

Evaluation of proposed treatment plans may go through the followingsteps: 1) feature calculation; 2) classification and quality estimation;and 3) QA review. Retrospective plans that were classified as clinicallyunacceptable may be used to assess the ability of the algorithm tocapture known errors and quantify expected deviations from clinicalpractice.

In some examples, the automated QA review process may be integrated withexisting clinical QA functionality by adding the following components:i) an analysis mechanism to provide feedback to the team quantitativelydetailing the underlying classification results; ii) a plan review andapproval system that may be combined with a comprehensive DICOM-RT basedimage and visualization platform to provide a complete integratedsolution; and iii) an interface to the DICOM-RT database that may beaccessed by the radiation treatment machine (e.g., linear accelerator)for actually programming the delivery of radiation the patient is toreceive in order to report any inconsistencies between databases,potentially contributing to errors in treatment delivery and patientsafety.

Data verification may play a role in RT. Plans may be exported into andout of various different software packages, and/or transmitted acrossnetworks. In some cases bit level comparisons of the plans may beappropriate, but when plans are exported from different softwarepackages this may not be appropriate. For example, at one point duringthe process the order of the beams could be changed in the DICOM file.This does not represent any change at all to treatment quality, but maychange the bit-level comparison results, producing an erroneous resultthat the two plans are different. In some examples, the disclosedmethods and systems may enable data verification at a higher level.

The inputs into the example of FIG. 1 have been simplified forillustrative purposes and may include: treatment planning data, imagingdata and regions of interests (if delineated, either manually,semi-automatically or completely automatically). Other sources of datanot represented in the figure may include previous outcomes data,including treatment related toxicity, survival data and recurrence data;data related to treatment delivery such as specifications of thetreatment delivery unit; or any other data pertinent to patienttreatment. Any patient information or plan feature useful for aphysician to make an accurate decision about the treatment plan may beincluded, and that information may be considered jointly as opposed toindependently [23].

In the example shown, the automated QA process may include variousinterconnected sub-components, including:

ROI classification and quality estimation algorithm (A),

treatment plan classification and quality estimation algorithm (C),

automated segmentation algorithms (D)

data comparison/integrity algorithm (J),

source of error algorithm (H),

visualization platform (I),

review and approval platform (I)

In the training phase, labelled ROIs and treatment plans may be used totrain an algorithm for ROI and plan classification and qualityestimation. This algorithm may be further broken into sub-componentspertaining to ROI classification and quality assessment, planclassification and quality assessment, and finally an integrated ROI andplan algorithm (blocks K, L and M, in FIG. 1). The ROI classificationalgorithm may be trained on a database of example ROIs with known labels(e.g., heart, lung, etc.). The treatment plan classifier may besimilarly trained on a database of example plans with known class.

Evaluation of a proposed treatment plan is now discussed. Once theproposed treatment plan, medical images, and ROIs are prepared the nextphase may be to automatically classify the data and provide estimates ofplan and ROI quality using the learned components and classifiers fromthe training phase. The basic pipeline may include the following:

Images may be first acquired. CT is predominately used in RT fortreatment planning although other imaging modalities can also be usedinstead, or in addition to augment the information for treatmentplanning.

From the desired imaging datasets, ROIs may be delineated manually,semi-automatically or completely automatically. This task may be done byeither the radiation oncologist or the planner. The treatment planner(dosimetrist) may then generate a treatment plan including the imagedataset(s), ROIs, defined anatomical points to facilitate treatmentplanning, treatment beams to treat the patient, the prescription dose toestablish the amount of radiation the target is to receive, and othersuch aspects.

The generated treatment plan may be then subject to technical andclinical review as part of the RT QA process. A medical radiationphysicist and the radiation oncologist may review the plan and determinethe clinical applicability of the plan.

The ROIs may be parsed and features extracted for the purposes ofclassifying each ROI in the treatment plan (block A). For each ROI,estimates of the quality may be provided for review and to establish theconfidence with which the ROI corresponds to a known ROI class (blockB). This may address both labelling errors (e.g., the planner labelledthe heart ROI as lung) and quality errors (e.g., the heart ROI is verypoorly contoured and contains a lot of lung, spinal column, etc.). Theseerrors can negatively affect treatment planning. In different examples,this step may take place before an initial plan is generated, or after,for example depending on the desired work flow of the target hospital.

The treatment plans may be parsed and features extracted for thepurposes of classifying the treatment plan with respect to the knowntreatment plan classes (block C). As previously noted, this step canoptionally take place after the planner has reviewed ROI qualityestimates to ensure there are no errors before generating the plan.

Automated segmentation algorithms may be run to ensure that a minimumnumber of ROIs exists in each treatment plan (block D). The automatedsegmentation may generate ROIs automatically using standard segmentationalgorithms and the features from these ROIs may be extracted using themethods of the ROI classification algorithm (block E). For example, theexternal outline of the patient may always be segmented in addition tothe lungs (e.g., in thorax images), the pelvis (e.g., in abdominalimages) and the skull (e.g., in head images). The features from theseROIs may then be added to the classification algorithm to aid in planclassification and help ensure there is consistent data used for theclassification process (block F).

An integrated plan and ROI algorithm may then prepare the finalestimates of plan and ROI quality estimates using information from theplan, the ROIs, and any additional automated segmentations available(block D). Similar to the ROI classification and quality estimationalgorithm, plan and ROI quality estimates may be generated from theintegrated classification algorithm. This may also provide a confidencefor the selected treatment plan class (block G).

In addition to providing an overall plan and ROI quality estimates, theexample method may provide estimates of what caused the low qualityscore. This may help give the planner an idea of what to change. Thismay take place in the quantitative feature analysis box (block H). Anymanually desired checks on the plan may also take place at this stage,ensuring, for example, if there is a known clinical tolerance for ahealthy organ (e.g., the spinal cord dose must be less than a certaindose or the target must be covered by a dose greater than or equal to agiven prescription dose).

In some examples, the output may also providing indication of qualityestimates for the plan and/or ROI that fall in a particular value range,such as being particularly high, and what caused the high qualityestimates. This may provide useful information to the user that certainparameters should be kept unchanged.

The plan and ROI quality estimates may be incorporated into the QAprocess in various ways such as:

The automated system may not be used and the RT team may render aclinical decision using only the visualization system alone (block K).

The system may be used to augment the current clinical process andprovide the RT team with a review of data from the feature extractionand with confidence that the plan belongs to a particular plan class andachieves the desired clinical criteria (block L).

The system may be used to render a clinical decision based on thetreatment plan class that has the highest confidence. From this point,the clinical criteria for that plan class may be used for furtheranalysis without requiring input from the RT team (block M).

Optionally, the system may enable a direct comparison between treatmentplan features that exist in the treatment plan and the same featuresthat would be present in the record, and may verify a database accessedby the radiation treatment machine (e.g., linear accelerator) foractually programming the delivery of radiation the patient is to receive(block J).

An example method for evaluating the quality of a proposed treatmentplan is illustrated in FIG. 18. This method may be carried out using anysuitable computing system, such as the example system of FIG. 2.

At 1805, a proposed treatment plan is obtained (e.g., automatically,from another system or manually entered). The proposed treatment planmay define RT treatment for at least one treatment site. A set ofpatient data (e.g., including patient features, such as a patientcharacteristic, a patient history, a patient diagnosis, and an imagedfeature, as discussed below) for the patient may also be obtained. Theremay also be one or more patient treatment requirements (e.g., limitationto permissible radiation dose, based on institutional guidelines) thatmay or may not be specific to the patient.

At 1810, the proposed treatment plan may be automatically characterizedaccording to one or more predefined features to determine the treatmentplan characterization.

Characterizing the proposed treatment plan may include determining atreatment plan class for the proposed treatment plan according to one ofa plurality of predefined treatment plan classes. This may be carriedout using an automated classification algorithm, as discussed below. Insome examples, the automated classification algorithm may be developedby machine learning (e.g., based on Random Forest techniques, asdiscussed below) using historical data. Where machine learning is used,such learning may be ongoing, as additional historical data becomesavailable, such that the classification of a given treatment plan may berefined over time, for example.

Characterization of the proposed treatment plan, such as classifying theproposed treatment plan, may be based on determining the similarity(e.g., as calculated through distances) of features of the proposedtreatment plan to features of known predefined treatment plan classes orcharacteristics.

Characterizing the proposed treatment plan may also include determininga quality estimate that the proposed treatment plan belongs to a giventreatment plan characterization, based on predefined expected featuresof the given treatment plan characterization. The proposed treatmentplan may be characterized according to various treatment plan features,for example, including one or more of: an anatomical site, a tumourhistology, a prescription dose, a treatment technique, and a treatmentintent.

Optionally, at 1815, ROI data may be obtained (e.g., from another systemor manually entered). The ROI data may delineate at least one ROI in aset of image data, where the image data may be of the treatment site andmay be included as part of the patient data. The ROI may be delineatedmanually, semi-automatically or fully automatically. In some examples,the ROI may be automatically segmented from the image data by thesystem, as part of the example method.

If there is ROI data, at 1820 the ROI may be automatically characterizedaccording to one or more predefined features to determine at least oneROI characterization. Similarly to 1810, characterizing the at least oneROI may include determining at least one ROI class respectively for theat least one ROI, according to one of a plurality of predefined ROIclasses, using the same or different automated classification algorithm.For example, the automated classification algorithm for classifying theROI may be developed by machine learning using historical data. Theautomated classification algorithm may be based on determiningsimilarity of features of the ROI to features of a predefined ROI class.

The ROI characterization may be determined based on shape and densityvalue of a given ROI, for example. The ROI may be characterizedaccording to ROI features including one or more of: anatomicalcorrespondence, tumours, dosage, regions to avoid, regions for doseevaluation, reference structures and structures to facilitate treatmentplanning, for example, and as discussed further below.

At 1825, a quality estimate (or confidence level) for the proposedtreatment plan may be calculated. The quality estimate may be calculatedas a confidence value, a simple pass/fail determination, or any othersuitable estimate of quality. The calculation may also includedetermination of how well certain plan features match with expected plancharacteristics and may include generation of error and/or warnings ifthe quality estimate is below a certain threshold value.

The quality estimate may be calculated by evaluating the proposedtreatment plan according to one or more rules defining expectedrelationships between the treatment plan characterization and one ormore of: one or more plan features, and one or more patient featuresdefined in the set of patient data.

The rule(s) may be defined (e.g., predefined manually) ormachine-learned based on historical data of historical treatment plans,as discussed further below. Where machine learning is used, suchlearning may be ongoing, as additional historical data becomesavailable, such that the rule(s) may be refined over time, for example.The historical data of historical treatment plans may include treatmentoutcome data. Rules defining expected relationships may include, forexample, one or more of: historical suitability of a given treatmentplan characterization for historical patients; historical treatmentoutcome of a given treatment plan characterization for historicalpatients; historical treatment plans for a specific patient; historicaltreatment outcomes for the specific patient; a mathematical function;and a general rule governing treatment plans irrespective of thetreatment plan characterization and irrespective of the patient data;among others.

Where there is ROI data, calculating the quality estimate for theproposed treatment plan may also include evaluating the proposedtreatment plan according to one or more rules defining expectedrelationships between the treatment plan characterization and the atleast one ROI characterization. Where there is ROI data, calculating thequality estimate may include calculating a quality estimate that a givenROI belongs to a given ROI characterization, based on predefinedexpected features of the given ROI characterization.

Where there is one or more patient treatment requirements, calculatingthe quality estimate may include evaluating whether the treatment plancharacterization satisfies the patient treatment requirement(s).

At 1830, output indicating the calculated quality estimate may beprovided. For example, a report may be generated and displayed to theuser on a display device (e.g., display screen of any suitable computingdevice) and/or may be printed. The output may be provided via a webportal. For example, a user may access the example system through aseparate client device (e.g., a mobile computing device, such as atablet or a smartphone) via a web-accessible portable provided by thesystem. This may enable greater mobility and flexibility for userinteraction with the system. The report may be stored for furtherprocessing and/or future reference in an internal or external memory ofthe system, for example.

The output may include one or more suggestions for modifying theproposed treatment plan (e.g., modifying ROI delineation, beamconfiguration, etc.) in order to improve the quality estimate. Theoutput may also include one or more features of the treatment plancharacterization that is relevant to the quality estimate. For example,if the proposed treatment plan was found to have low quality because ofa mislabelled ROI, the output may include indication of the mislabelledROI and that this error contributed to a low quality estimate.

Similarly, the output may include one or more suggestions for whatparameter (e.g., a particular ROI or a particular beam geometry) shouldbe kept unchanged. For example, a parameter that was determined to havecontributed to an increase in the quality estimate may be indicated as aparameter that should be kept unchanged.

The output may include the calculated quality estimate and/or planfeatures and/or ROI features for the proposed treatment plan, ascompared to the quality estimate and/or features from the historicalplan class and/or ROI class of interest.

Thus, the example method for evaluating the proposed treatment plan mayenable estimation of the quality of a proposed treatment plan based onan integrated consideration of plan features optionally together withROI features, patient features, and other considerations (e.g.,treatment requirements), for example.

An example method for evaluating the quality of a delineated ROI isillustrated in FIG. 19. This example method may be similar to theexample method of FIG. 18, and may be carried out using any suitablecomputing system, such as the example system of FIG. 2.

At 1905, a set of ROI data may be obtained (e.g., entered manually,obtained from another system or automatically segmented by the systemfrom image data). The ROI data may delineate at least one ROI for atleast one treatment site in a set of image data. A set of patient data(which may include one or more patient features, as discussed furtherbelow) for a patient may also be obtained. One or more patient treatmentrequirements may also be obtained, as discussed above.

At 1910, the ROI may be automatically characterized according to one ormore predefined features to determine at least one ROI characterization.This may be similar to 1820 discussed above (e.g., using an automatedclassification algorithm or other characterization techniques), and suchdiscussion need not be repeated here. Characterizing the ROI may includedetermining a quality estimate that the ROI belongs to a given ROIcharacterization, based on predefined expected features of the given ROIcharacterization. The ROI may be characterized according to one or moreROI features including one or more of: anatomical correspondence,tumours, dosage, regions to avoid, regions for dose evaluation, and areference structure, among others, as discussed further below.

Optionally, at 1915, a proposed treatment plan may be obtained (e.g.,entered manually or communicated from another system). The proposedtreatment plan may define treatment for the at least one treatment site,as discussed above.

If there is a proposed treatment plan, then at 1920 the proposedtreatment plan may be automatically characterized according to one ormore predefined features to determine a treatment plan characterization.This may be similar to 1810 discussed above (e.g., using an automatedclassification algorithm or other characterization techniques; andoptionally including determination of a quality estimate for theproposed treatment plan as discussed above), and such discussion neednot be repeated here.

At 1925, a quality estimate may be calculated for the ROI by evaluatingthe ROI according to one or more rules. The rules may define expectedrelationships between the ROI characterization and one or more of: oneor more ROI features, and one or more patient features defined in theset of patient data. Similar to 1825 discussed above, the rules may bedefined (e.g., predefined manually) or machine-learned based onhistorical data of historical ROIs. Rule(s) may define expectedrelationships, for example including one or more of: historicalsuitability of a given ROI characterization for historical patients;historical treatment outcome of a given ROI characterization forhistorical patients; historical ROIs for a specific patient (this mayinclude relationships based on ROI features, for example that heart ROIsare associated with a round shape, and this may be dependent on thecontext of the patient's image data); relationships between differentROIs within a specific patient (e.g., smaller lung ROIs may be expectedto be associated with a smaller heart ROI); historical treatmentoutcomes for the specific patient; a mathematical function; and ageneral rule governing ROIs irrespective of the ROI characterization andirrespective of the patient data; among others.

The quality estimate may be calculated as a confidence value, a simplepass/fail determination, or any other suitable estimate of quality. Thecalculation may also include determination of how well certain ROIfeatures match with expected ROI characteristics and may includegeneration of error and/or warnings if the quality estimate is below acertain threshold value.

Where there is a proposed treatment plan, calculating the qualityestimate may include evaluating the ROI according to one or more rulesdefining expected relationships between the ROI characterization and thetreatment plan characterization.

Where there are one or more patient treatment requirements, the qualityestimate may be calculated based on evaluating the ROI according to oneor more rules defining expected relationships between the ROIcharacterization and the patient treatment requirement(s).

At 1930, output indicating the calculated quality estimate may beprovided, similar to 1830 above (e.g., output may include guidelines forimproving the quality estimate, may include information relevant to thequality estimate, may be provided in the form of a report and/oraccessible via a web portal, etc.). Such discussion need not be repeatedhere in detail.

The above description may be implemented using the example system ofFIG. 2. For example, the example system may include: 1) Theclassification and quality estimation algorithms described aboveproviding quantitative analysis of the each treatment plan and ROI inthe plan (e.g., to carry out block H in FIG. 1; and blocks 1810 and 1820in FIG. 18); and 2) the visualization and review platform which mayprovide a user with the ability to interrogate treatment plans and mayprovide a mechanism for review to render a clinical decision as to theclinical acceptability of the plan (e.g., to carry out block I in FIG.1; and block 1830 in FIG. 18).Automated Treatment Planning: Overview of Example Method and System

As discussed above, the present disclosure enables automated treatmentplanning without the requirement for delineated ROIs in the patient'simage data. However, it may be the case that ROIs have been delineatedin the historical data stored in the system database. Additionally, thedisclosed method may still be carried out where the image data doinclude ROIs.

If, for a particular treatment plan class to be planned, ROIs have beendelineated in the training database then:

-   -   1) If there are ROIs for the novel patient image data, a dose        distribution can be inferred based on ROI specific features and        optionally also non-ROI specific features such as texture,        density etc in the image as it relates to the dose distribution.    -   2) If there are no ROIs for the novel patient image data, a dose        distribution can be inferred based on features that are not        dependent on the ROIs.

If, for a particular treatment plan class to be planned, ROIs have notbeen delineated in the training database (e.g., in the case of simpletreatment plans that are palliative in nature) then:

-   -   1) If there are ROIs for the novel patient image data, a dose        distribution can be inferred based on features that are not        dependent on the ROIs as there are no ROIs in the treatment plan        class to calculate ROI-based features.    -   2) If there are no ROIs for the novel patient image data, a dose        distribution can be inferred based on features that are not        dependent on the ROIs.

In some examples, the disclosed method may be advantageous overconventional automated planning approaches in one or more ways, such as:

The entire dose distribution for the patient may be planned.

ROIs may not be required to infer the dose distribution.

Where ROIs are present in the patient image data, ROIs may be used todrive the inferred dose distribution using ROI-based features as theyrelate to the dose distribution.

Dose-volume objectives may not be required.

A dose distribution may be inferred for even simple cases that would notrequire an optimization engine for conventional treatment planning.

An example method for generating a proposed treatment plan is shown inFIG. 20a . This method may be carried out using any suitable computingsystem, such as the example system of FIG. 2. FIG. 20b , discussedfurther below, illustrates further details of the example method.

At 2005, a set of patient data is obtained, including at least a set ofimage data for the patient. The patient's image data may or may notinclude delineated ROIs.

At 2010, a treatment plan class is determined. A treatment plan class isa broad categorization or characterization of the intended treatment. Aplurality of treatment plan classes may be predefined (e.g., based oncategorization of historical data). Each predefined treatment plan classmay define one or more treatment plan features (e.g., tissues to avoidor dosage to use) for a respective treatment site. For example, atreatment plan class may be a breast treatment or a prostate treatment.The treatment plan class is used to determine the rules to be used forgenerating the proposed treatment plan, as discussed further below.Determination of the treatment plan class may be carried out at leastbased on the image data (e.g., using an automatic classificationalgorithm). Additionally or alternatively, the determination of atreatment plan class may be based on user input. For example, a usermight provide input indicating that the treatment is to be a breasttreatment; alternatively, the image data may be automatically analyzedto determine that the image is that of a breast with a tumour andaccordingly the treatment should be a breast treatment.

At 2015, a proposed dose map is calculated. This calculation may becarried out using inference techniques, as discussed further below. Theproposed dose distribution (also referred to as a dose map) may definethe proposed dosage over a volume of the image data (e.g., on avoxel-by-voxel basis). The proposed dose map may be calculated inaccordance with a set of rules defining relationships between applieddosage, the treatment plan class and at least one feature of the imagedata. These rules may include rules generated by computer learning basedon historical data, may include mathematical functions, and may includehard rules that are manually inputted, for example.

The proposed dose map may be inferred on a voxel-by-voxel basis. Eachvoxel may be characterized based on one or more appearance features(e.g., density). Then, the set of rules may be used to relate the voxelfeature(s) to the treatment plan class and optionally other patientfeatures. For example, for a breast treatment plan class, a voxel may becharacterized as breast tissue, and the patient data may indicate thatthe patient is a 24 year old. The set of rules may then include a ruleindicating that, for a breast treatment plan, breast tissue for a youngadult should receive a specified dose. Such a rule may have been learnedbased on historical data showing that similar patients undergoing breastRT received similar dosages for breast tissue, for example.

At 2020, one or more treatment plan parameters are determined in orderto achieve the proposed dose map. The treatment plan parameters may bedetermined in accordance with another set of rules, which may includeoptimization algorithms, and may include manually inputted rules, forexample. The treatment plan parameters may include beam parameters(e.g., beam shapes, beam intensities and beam modulation), for example,and may be determined in accordance with beam optimization algorithms.

At 2025, the proposed treatment plan is generated and outputted. Theproposed treatment plan includes the one or more determined treatmentplan parameters. The proposed treatment plan may be outputted visually(e.g., displayed on a screen) and/or may be transmitted to a databasefor future consideration.

FIG. 20b illustrates an example method for automated treatment planningin greater detail.

At 2052, the set of patient data, including the set of image data, suchas CT imaging data, is obtained. The set of patient data may include,for example, patient features, such as a patient characteristic, apatient history, a patient diagnosis, and an imaged feature.

Optionally, in addition to the patient data, there may also be one ormore patient treatment requirements (e.g., limitation to permissibleradiation dose, institutional guidelines, clinical protocol, clinicaltrial criteria) that may or may not be specific to the patient.

Optionally, ROI data may be obtained (e.g., from another system ormanually entered). The ROI data may be included in the image data or maybe a separate set of data. The ROI data may delineate at least one ROIin the set of image data. The ROI may be delineated manually,semi-automatically or fully automatically. In some examples, the ROI maybe automatically segmented from the image data by the system, as part ofthe example method. ROIs may be characterized using an automatedclassification algorithm, for example as discussed above with respect tothe automated QA method and system. In some examples, the automatedclassification algorithm may be developed by machine learning (e.g.,based on Random Forest techniques, as discussed below) using historicaldata.

At 2054, the treatment plan class is set. The treatment plan class maybe one of several predefined treatment plan classes. The treatment planclass may set be based on manual input from the user (e.g., the user maymanually enter input specifying the desired treatment plan class), basedon an electronic medical record (e.g., the patient data may bereferenced to a patient recording including information about thenecessary treatment for the patient) and/or based on automatedclassification of the image data (e.g., using any ROIs, features of theimage data, such as the pulse sequence used and/or physiologicalfeatures in the image) and/or the imaging protocol used.

If the treatment plan class is automatically determined, thisdetermination may be based on a comparison of the inputted patient datawith historical data, to determine the similarity of the inputtedpatient data to previous patients and the treatment plans used. Thisautomatic classification may be performed using machine learning. Wheremachine learning is used, such learning may be ongoing, as additionalhistorical data becomes available, such that the classification of agiven treatment plan may be refined over time, for example. Determiningthe treatment plan class may be based on determining the similarity(e.g., as calculated through distances) of features of the image data tofeatures of historical image data which have historical treatment plans.

If the treatment plan class is determined based on the electronicmedical record system and/or the oncology information system and/orbased on automated classification, a prompt may be generated requestingthe user to approve or reject the determined treatment plan class.

A treatment plan class would define, for example, the site of treatmenti.e. prostate, breast, lung, and/or the histology of the disease i.e.non-small cell, mesothelioma, and/or the prescribed dose i.e. 7800 cGyin 39 fractions, 800 cGy in 1 fraction, and/or the treatment techniquei.e. static beams, IMRT, VMAT, and/or the clinical intent i.e. radical,palliative, pain management, The proposed treatment may define RTtreatment for at least one treatment site.

At 2056, after a treatment plan class has been set, it is determinedwhether there is ROI data.

Optionally, at 2058, if there is ROI data, the ROI may be manually orautomatically characterized according to one or more predefined featuresto determine at least one ROI characterization, for example similarly tothat described above for the automated QA method and system. In someexamples, where ROI data is not provided as input data, ROI data may beautomatically generated by automatically segmenting one or more ROIsfrom the image data. The ROI data, whether provided as input orautomatically generated, may be evaluated for quality, for example asdescribed above.

At 2060, one or more features that will serve as the basis forgenerating the proposed treatment plan may be calculated. The featuresthat are calculated for the proposed treatment plan may be based on thedetermined treatment plan. For example, the features that would becalculated for a given set of patient data may be based on the set ofimage data (and ROI data if available).

The calculated features may include features extracted from the dataprovided at 2052 (e.g., images, ROIs, patient history, etc.). The set offeatures may or may not include features specific for the treatment planclass. Features of the image data may include non-ROI features, such ascharacterization of image portions (e.g., voxels) or the entire image.For example, the image data may be characterized according to thetexture and/or intensity of portions of the image.

Intensity refers to the intensity (or magnitude) of an image signal in agiven portion of the image. In the case of CT images, this quantity isalso referred to as density. Texture is a feature extracted from animage by examining the intensity patterns. So, for example, an imageregion full of small circles would have a different texture than animage region full of small squares. Such characterization may enablecharacterization of the intensity patterns in an image portion or entireimage. Techniques for characterizing intensity of image data include,for example, textures, localized histograms, gradients, or otherpattern-recognition techniques.

This characterization may be used to measure, estimate and/or representthe different anatomies and geometries of image data. For example, thismay enable a heart-like region in the image that is near the lung to becharacterized as being different from a heart-like region in the imagethat is in the middle of the heart.

Use of non-ROI image features may complement ROI data, where available.For example, even if a heart ROI has been delineated in the image data,non-ROI characterization may still be used to characterize where a givenvoxel is within the heart.

At 2062, the features calculated in 2060 may be correlated withhistorical data (e.g., from a database of historical image data,historical ROI data and/or historical dosage data). This correlation maybe carried out based on the treatment plan class, so that the mostappropriate historical data is referenced.

Based on the correlation, at 2063, a proposed dose map (also referred toas a proposed treatment dose distribution) may be calculated. Theproposed dose map may be calculated as a scalar voxel-by-voxel dosedistribution corresponding to the input image data, or at a lower orhigher spatial resolution, and may or may not include a confidenceestimate or probability distribution at each location specifying theconfidence of the estimation and/or the probability of different doselevels. The calculation may also include determination of how wellcertain image and/or ROI features match with expected characteristicsand may include generation of error and/or warnings if the quality ofthe image data at a given location is estimated to be insufficient todetermine dose at that given location.

The proposed dose map may be tailored according to a first set of one ormore rules defining expected relationships between applied dosage, thetreatment plan class, and one or more patient features defined in theset of patient data.

The rule(s) may be defined (e.g., predefined manually) ormachine-learned based on historical data of historical treatment plansand the associated dose distribution of the treatment plans in thetreatment plan class. Where machine learning is used, such learning maybe ongoing, as additional historical data becomes available, such thatthe rule(s) may be refined over time, for example. The historical dataof historical treatment plans may include treatment outcome data. Rulesdefining expected relationships may include, among others, one or moreof:

-   -   historical suitability of a historical dose map for the        treatment plan class (e.g., inferring the dose distribution for        the patient based on historical treatment plans and their        associated dose distributions);    -   historical treatment outcome of a given dose map for historical        patients (e.g., considering the known outcomes of historical        dose distribution to tailor the proposed dose map for achieving        a desired outcome);    -   historical dose maps for a specific patient (e.g., where a        previous treatment plan for the same patient is stored in the        database, the same or slightly altered dose map may be        suitable);    -   historical treatment outcomes for the specific patient (e.g.,        where a previous treatment outcome for the same patient is        stored in the database, the dose map may be copied from the        historical data if the currently desired treatment outcome is        the same);    -   a mathematical function (e.g., a regression algorithm or        function relating dose values to image features, such as f(x)        where the output is a dose value and x is a set of image        features); and    -   a general rule governing dose maps irrespective of the treatment        plan class and irrespective of the patient data (e.g., the        patient cannot receive more than a specified dosage), which may        be entered manually.

The proposed dose map may have an associated voxel-by-voxel variability(distribution of possible doses), which can be used to tailor the dosedistribution on a plan class by plan class basis. Such variability maybe expressed as a proposed range, a proposed likelihood distribution ora proposed value with confidence measure, for example. The specific doseto apply to a voxel may be selected from the dose range according to athird set of rules. For example, for a given plan class, the inferredvoxel-by-voxel dose may be set to the median value for the distributionof inferred values and for another plan class the inferredvoxel-by-voxel dose may be set to the maximum value for the distributionof inferred values, corresponding to the requirements of the particulartreatment plan class.

The proposed dose map may provide a voxel-by-voxel mapping of proposeddose distribution over the image data. The voxel resolution of theproposed dose map may or may not correspond to the resolution of theimage data. For example, a 256×256 proposed dose map may be generatedfor a 512×512 image, or vice versa.

In some examples, the proposed dose map may be outputted (e.g.,displayed on a display device, such as a computer screen) forvisualization by the user. This may be, for example, a voxel-by-voxeldose map superimposed on the image data. In some examples, thevisualization of the proposed dose map may include visualization of aconfidence measure (where the proposed dose map has variability, asdiscussed above) of each voxel of the proposed dose map, for example bycolor-coding the proposed dose map according to the confidence measure.

The outputted visualization may be provided with a user interface toenable the user to manually modify proposed dosages and/or imagefeatures, such as delineated ROI(s), used to calculate the proposed dosemap. For example, the user interface may provide the user with editingtools (e.g., a paintbrush tool) to increase or decrease the proposeddose for voxel(s) of the proposed dose map and/or to add, remove and/ormodify ROI(s) in the image. After any changes are made, the user mayconfirm the changes and the method may return to one or more of theabove-mentioned steps (depending on the extent of the user's changes) torecalculate the proposed dose map.

Optionally, at 2064 and 2065, automated QA may be performed on theproposed dose map. This may be carried out using the automated QA methoddescribed above, for example. If the QA fails (e.g., the confidencelevel is below an acceptable threshold) the user may be prompted tomanually change the proposed dose map (e.g., using the visualization anduser interface described above) at 2066.

At 2067 and 2068, the proposed dose map may be used to generatetreatment parameters for a proposed treatment plan, in accordance with asecond set of one or more rules. The second set of rules may optionallyinclude optimization algorithms for optimizing beam fluence 2067, forexample. The second set of rules may include, among others, one or moreof:

-   -   historical treatment parameters for a given planned dosage        (e.g., three beams are used to reach given dose in dense        tissue);    -   historical treatment parameters for a given treatment plan class        (e.g., breast treatments always involve three beams);    -   a mathematical function (e.g., a regression algorithm or        function relating beam angle to image features); and    -   a general rule governing treatment parameters irrespective of        the proposed dose map (e.g., defining the number of beams, the        beam orientations, the beam intensities, the beam shapes, and        the modulation complexity of the beams as specified by the        resolution of the beam fluence generated via optimization        algorithms).

Optionally, at 2069, a quality estimate (or confidence level) for theproposed treatment plan may be calculated. The quality estimate may becalculated as a confidence value, a simple pass/fail determination, orany other suitable estimate of quality. That calculation may resultdirectly from the dose map inference process and/or the treatment plangenerated following a suitable optimization method which defines thebeams and/or beam shapes and/or beam intensities and/or beam modulation.The calculation may also include determination of how well certain planfeatures match with expected plan characteristics and may includegeneration of error and/or warnings if the quality estimate is below acertain threshold value. If QA fails (e.g., the confidence level isbelow an acceptable threshold), the user may be prompted to manuallychange the proposed treatment plan at 2070, to change the input data, tochange the proposed dose map and/or to change optimization parameters.

At 2071, the expected dose distribution for the proposed treatment planis calculated. Verification may be performed at this point. Verificationmay include, for example, checking whether the expected dosedistribution matches with the proposed dose map, checking whether theexpected dose distribution falls within acceptable dose guidelines,and/or checking whether the expected dose distribution is similar tohistorical dose distributions.

At 2072, if verification fails, the proposed dose map may be changed at2073 (e.g., manually or automatically, such as using an iterativemethod), and the method returns to 2068 without generating the proposeddose map.

At 2072, if verification fails, the user may be prompted to manuallychange the input data at 2073 (e.g., input a different treatment planclass, add or remove ROIs, etc.), and the method returns to 2054 togenerate the proposed dose map.

At 2074, if verification fails because the calculated expected dosedistribution does not match the proposed dose map, the proposedtreatment parameters may be changed and/or the beam optimization may berecalculated using different optimization parameters, and the methodreturns to 2068.

At 2078, if verification is successful, then the method proceeds to2080.

Optionally at 2072, 2074 and 2078, a quality estimate (or confidencelevel) for the calculated expected dose distribution may be calculatedusing the example automated QA method described above. The qualityestimate may be calculated as a confidence value, a simple pass/faildetermination, or any other suitable estimate of quality. Thatcalculation may result directly from the dose map inference processand/or the treatment plan optimization process and/or the treatment plangeneration process, and may follow a suitable optimization method whichdefines the beams and/or beam shapes and/or beam intensities and/or beammodulation. The calculation may also include determination of how wellcertain features of the proposed treatment plan match with expected plancharacteristics (e.g., based on historical data for the treatment planclass) and may include generation of error and/or warnings if thequality estimate is below a certain threshold value. If QA fails (e.g.,the confidence level is below an acceptable threshold), the user may beprompted to manually change the calculated expected dose distribution at2070 and/or 2073, to change the input data, to change the proposed dosemap and/or to change optimization parameters, for example.

At 2080, the proposed treatment plan is outputted (e.g., presented to auser visually on a screen). The output may be provided via a web portal.For example, a user may access the example system through a separateclient device (e.g., a mobile computing device, such as a tablet or asmartphone) via a web-accessible portable provided by the system. Thismay enable greater mobility and flexibility for user interaction withthe system. The proposed treatment plan may also be transmitted to adatabase for future consideration and/or may be added to the historicaldata.

Optionally, output indicating the calculated quality estimate may alsobe provided. For example, a report may be generated and displayed to theuser on a display device (e.g., display screen of any suitable computingdevice) and/or may be printed. The report may be stored for furtherprocessing and/or future reference in an internal or external memory ofthe system, for example.

The output may include any output of the automated QA system, and/or anestimate of confidence at each voxel of the dose map indicating wherethere is uncertainty or ambiguity as to whether or not to irradiate apatient at that location.

Thus, the example method generating the proposed treatment plan may alsoenable estimation of the quality of a proposed treatment plan based onan integrated consideration of plan features optionally together withROI features, patient features, and other considerations (e.g.,treatment requirements), for example.

In some examples, the disclosed methods and systems may use machinelearning to build an automated planning method for treatment planning.Imaging and treatment planning data may be grouped according totreatment site and treatment technique. Relevant features (e.g., asshown in FIG. 21) for inferring the dose map may include, for example,image texture [38], image intensity, image position, position insidepatient [39] and position inside ROI.

Features may be extracted from historical training data, and used toconstruct the learning algorithms and train the treatment planningmodel. Additional features, not found directly in the RT images and/orROIs, may be automatically generated by applying segmentationalgorithms, which can delineate relevant organs in the RT image dataset.

Automated inference of the proposed dose map may include the followingsteps: 1) image segmentation; 2) feature calculation; 3) plan generation4) quality estimation; and 5) review. Retrospective plans that wereclassified as clinically unacceptable may be used to assess the abilityof the algorithm to generate plans for patients who previously failedthe standard treatment planning clinical practice.

In some examples, the automated dose mapping process may be integratedwith existing clinical treatment planning functionality by providing anestimate of the plan that can be refined, or used as a starting pointfor the standard clinical practice.

The inputs into the example of FIG. 20b have been simplified forillustrative purposes and may include: existing treatment planning data,imaging data and regions of interests (if delineated, either manually,semi-automatically or completely automatically). Other sources of datanot represented in the figure may include previous outcomes data,including treatment related toxicity, survival data and recurrence data;data related to treatment delivery such as specifications of thetreatment delivery unit; or any other data pertinent to patienttreatment. Any patient information useful for a physician to make anaccurate decision about the treatment plan may be included, and thatinformation may be considered jointly as opposed to independently [23].

In the example shown, the automated treatment planning process mayinclude various interconnected sub-components, which may be similar tothe components described above for the automated QA process, including:

Manual, semi-automatic, or automatic segmentation,

ROI classification and quality estimation algorithm,

Dose map confidence and/or quality estimations algorithm,

visualization platform,

review and approval platform In the training phase, labelled ROIs andpreviously approved treatment plans may be used to train an algorithmfor automated treatment planning. The automated treatment planningalgorithm may be trained on a database of example plans with or withoutknown labelled ROI (e.g., heart, lung, etc.).

Automated treatment planning of a clinical RT treatment plan is nowdiscussed. Once the plan is generated the next phase might be furtherrefinement or review either manually or in an automatic setting (as inthe next section). The basic automated treatment planning pipeline mayinclude the following:

Images may be first acquired. CT is predominately used in RT fortreatment planning although other imaging modalities can also be usedinstead, or in addition to augment the information for treatmentplanning.

From the desired imaging datasets, ROIs may be delineated manually,semi-automatically or completely automatically. This task may be done bythe radiation oncologist or the treatment planner, for example. Theautomated segmentation may generate ROIs automatically using standardsegmentation algorithms and the features from these ROIs may beextracted using the methods of the ROI classification algorithm. Forexample, the external outline of the patient may always be segmented inaddition to the lungs (e.g., in thorax images), the pelvis (e.g., inabdominal images) and the skull (e.g., in head images).

The patient data is then processed into features which may, but are notlimited to, image-based features, ROI features, patient history, andother biomarkers.

In some examples, the algorithm takes as input one or more images havingelements which might include a single pixel at the fine-grain to groupsof pixels at the coarser grain. Based on features of the image,elements, patient history, treatment intent, etc., or any combinationthereof, the algorithm then establishes a correspondence (or matching)between elements in the input and elements in the database. Thecorrespondence, or mapping, can be obtained implicitly (e.g., usingmachine learning algorithms such as regression) or explicitly (e.g.,using image registration [40]). The correspondence can be betweenelements in a pair of patients (one patient to be planned, onehistorical patient), or to many patients (one or many patients to beplanned, to one or many historical patients). In the case of adaptiveplanning, the historical data can also include the patient's ownprevious treatments. The dose can then be calculated from the known doseto corresponding elements in the database, e.g. via averaging over thedose of the most similar elements.

Approaching the problem as one of establishing correspondence orsimilarity in this manner allows for the interchangeability ofregistration, regression, and classification algorithms. Thecorrespondence may be regularized spatially, as in the example ofdeformable image registration, to ensure a relatively consistent dosemap is defined for neighboring image elements, e.g. the dose to beapplied nearby voxels is relatively similar, or, for example, via aconditional random field (e.g., as used in the Conditional Random Forestalgorithm eq. (2) and eq. (11), replacing F with features from Table 9(FIG. 21) and c with dose-per-element).

The extracted features are then input into the learned dose predictionalgorithm, which outputs a dose estimate for each voxel in the inputimage. The dose estimate might be, for example, a maximum-posteriorestimate taken over a learned probability distribution of dose givenfeatures, or the output of a regression method [41, 42] or estimated viaatlas-based registration replacing the canonical atlas-segmentation withan atlas-dose map [43-45].

The dose estimate may include a probability distribution over all dosevalues that estimates the probability of a particular dose at that voxelgiven the observed features, or another form of confidence measure inthe predicted output, such as a confidence interval.

The generated treatment plan may be then subject to technical andclinical review as part of the RT process using standard approaches orusing the automated QA method described above.

In addition to providing an overall dose map and confidence and/orprobability estimate, the example method may provide estimates of whatcaused uncertainty in the dose distribution. For example, in the case ofextreme image noise the dose map will have a high degree of uncertaintyand the corresponding image features will be reported.

The generated dose map may be incorporated into the treatment planningprocess in various ways such as:

The method and system may be used to augment the current clinicalprocess and provide the RT team with a reference plan.

The dose map may be run through existing treatment planning systems togenerate a full plan for clinical treatment.

Optionally, the system may be used to automatically adapt (e.g., using adeformable register) the patient's planned treatment to a treatment dayimage, or subsequent treatment day by including the planned treatment(or previous treatments) in the learning database with or without theprevious plans from other patients.

The system of FIG. 2 may be used to implement the automated treatmentplanning system, and may include: 1) The automated dose map predictionalgorithm described above; and 2) the visualization and review platformwhich may provide a user with the ability to interrogate the dose map.The user may be able to edit the dose map manually, such as through theaddition of plan information such as adding ROIs, or automatically. Thedose inference or treatment optimization may then be updated to generatean updated calculated dose map.Example Algorithms

Various examples of the disclosed methods and systems may be implementedusing one or more of the example algorithms discussed below. Thealgorithms may be customized depending upon the specific application,i.e. automated QA or automated treatment planning. The algorithms forautomated QA may also be applied for automated treatment planning, forexample using the ROI classification algorithm to classify ROIs to beused for feature calculation as input to the treatment planningalgorithm. These are provided for the purpose of illustration only andare not intended to be limiting. The present disclosure is not bound byany theory or model discussed herein.

An example approach to perform the ROI and plan classification may be totreat them as separate classification algorithms, and rely on a thirdalgorithm to integrate the results. In principle, all three algorithmscould be the same exemplar classification algorithm (e.g. Support VectorMachines, Random Forest, K-means, etc.) [42, 46], or each algorithm canbe a customized classifier designed specifically for the problem at hand[47]. Customized classifiers may yield better performance (see exampleresults comparing classification of ROI in Table 6, Table 7 (FIGS.14a-14c and 15a-15c ); and treatment plan in Table 5 (FIG. 13a-13c )).One such example is the ROI classification algorithm implemented in[47]. The ROI classification algorithm may be customized for ROIs byincluding groupwise features calculated between the different ROIs in aplan, and ensuring that the assignment of class labels across all ROIsin the plan is consistent.

Similarly, for automated treatment planning, a canonical regressionalgorithm [41, 42, 46] can be used, or a customized technique can bedeveloped following that of the ROI classification algorithm [47].

For automated QA, the classifiers can be considered as blackboxes as inFIG. 1, blocks A and C. Each classifier may take as input a set offeatures, F, and may output a class label Y. Each classificationalgorithm may be augmented to further include a quality estimate, Q(F).For the case of ROIs, the quality estimate may be an estimate of eachROI's quality. For plans, the quality estimate may be an estimate of theoverall plan quality. The quality estimate may be generically referredto as a function Q(F), where the features, F, can even include the classlabel itself. Quality estimates can be made through density estimates[47] which may be the probability of observing a given set of ROIfeatures together with a specified ROI label, or the probability ofobserving a given set of geometrical patient features, in conjunctionwith particular beam geometry or dose distribution. This may capture,for example, the relationship that patients with large hearts typicallyhave a different beam configuration than patients with small hearts. Ifa small heart patient is assigned the beam configuration of a largeheart patient, the example system may return a very low likelihood ofthat being correct. The ability of the example system to jointlyconsider patient and plan information may be in contrast to conventionalalgorithms that considered only plan information [23].

For automated treatment planning, the regression algorithm, as well asthe optional ROI classification algorithm, may also be considered blackboxes in a similar fashion to automated QA. The algorithms are similar,where the automated treatment planning learns and makes use of, forexample, the probability of observing a given dose at a given image(e.g. CT, MRI, etc.) patch using learned descriptors of the patch, inaddition but not limited to optional ROI and patient history featuresfor a particular RT class. Modelling, for example, that in radicalbreast RT patches with a heart-like appearance, or heart-based ROIfeatures, are to be given no dose.

Different examples of the disclosure methods and systems can includedifferent classifiers and/or regression algorithms, and differentdensity estimation methods, or classifiers with built in estimates ofquality and/or density, or avoid classification and perform densityestimation directly on available features. Again, plan quality may bejudged automatically by jointly examining plan and patient features,including ROI features if available. Similarly, automated planning byjointly examining patient features from the images and/or other sources,e.g. ROIs, patient history, treatment intent, etc. Instead of learningdensities, distance metrics [49] between patients and distance metricsbetween plans may be learned or used, thereby matching a new patient tothe most similar patient in the patient database. For automated QA, theplan for the new patient can then be compared to the plans of the mostsimilar patients using calculated features and/or a learned distance.For automated treatment planning, the plan for the new patient can becopied from the existing similar patient, or fused from many similarpatients, using one of the regression and/or registration algorithms. Adensity can be directly assigned (e.g. Gaussian with mean zero andstandard deviation 1) or learned from the training data. These learneddistances may be considered features, which can be used in the disclosedmethods and systems just like any other feature discussed herein.

It is important to note that while the scale of the features, and items,processed can impact algorithm accuracy, it is variable and in no waylimiting of the proposed systems. Automated QA or treatment planning caninvolve features and algorithms from the fine level (e.g. featuresper-pixel) to the moderately coarse (e.g. feature per image-patch orper-ROI) to the coarse (e.g. features per patient). For example, inautomated treatment planning this could mean learning to predict dosefor a particular image patch, or simply copying and/or registering thedose map from the most similar patient in the database (anearest-neighbour method).

Classification may be relevant for applications to data mining, and whenthe plan class itself is used as a distinguishing feature for accuratequality estimation (for example, when identical patient features cancorrespond to two different treatment techniques).

Relationships that are known a priori (i.e., that do not requirelearning) may also be used. For example, examples of the disclosedmethods and systems may include manually encoded (hard-coded) rules,such as to check that patients under the age of 25 do not receive morethan a particular dose amount. Such manual encodings may be included inthe quantitative feature analysis box (block H).

In some examples, classifiers and/or density estimation techniques maybe replaced with regression algorithms. Regression algorithms may berelated to classifiers, but they typically seek to predict a continuousoutput as opposed to a discrete class label. For example, a regressionapproach may attempt to learn Q(F) directly from the example qualityestimates given for the training data. Similarly for automated treatmentplanning, the regression may be replaced with classification predictingdose or no dose, instead of a continuous dose value. These methods mayrequire known estimates of plan quality, whereas density estimationmethods may assume plan quality is proportional to how often acombination of plan and patient features are observed together.

Automated treatment planning can also be augmented by the methods andfeatures used for automated QA. Automated QA, as disclosed herein, mayenable planners to generate a plan, and then query the system to see ifit meets clinical standards in a real-time fashion, and that feedbackcan be used to tweak the plan. If the planner is missing a beam, forexample, the example system may report this to the user and the plannercan then add another beam. Similarly, by learning a joint distributionover plan features and patient features, known variables may beintegrated by the example system to provide the planner with the mostlikely values for unknown variables. For example, if the plan class, ROIlabels, and number of beams are all known, the example system maycalculate the most likely layout of beams. This may be thought of as theexample system determining the typical beam configuration for a givenkind of patient (e.g., defined by that patient's features in the system,such as organ geometry, etc.). The most likely beam configuration canthen be used to assist the automated treatment planning algorithm withinferring the dose map.

Without loss of generality, some example embodiments of each mainalgorithm are discussed below. For brevity, only classifier-basedembodiments are discussed for automated QA, as extension toregression-based methods is expected to be straightforward. Onlyregression-based examples are discussed for automated treatmentplanning. Each subsequent section is kept brief where details of thealgorithm are not the focus of this disclosure.

Example ROI Classifier Algorithm

An example of the ROI classifier is described as follows. At a highlevel the example classier may use shape and density features calculatedindividually for each ROI as well as features between different ROIs toestimate the classes of a new set of ROIs.

Each RT plan considered may include one or more ROIs, and acorresponding CT image. In practice there may be no set number ofmaximum ROIs that might appear in a plan. For example, a plan mightcontain a heart ROI and two lung ROIs, a heart ROI with no lung ROIs, oran anal canal ROI with a rectum ROI. Which ROIs can and cannot appeartogether fluctuates across different cancer treatment centres, and sothis relationship may need to be learned instead of set manually. CTimages are typically used almost universally for treatment planning asthey may provide a direct way to estimate the density of a given voxel,and therefore how much radiation dose will be delivered to a particularROI. However, the present disclosure may be implemented with othermodalities (e.g. MRI, cone-beam, etc.). An example of a suitableclassifier algorithm is presented in [48], and briefly discussed below.

The individual features of ROIs may be first outlined, and a RandomForest (RF) classifier may work from just those features. Inference maythen be performed using a learned prior distribution over which ROIclasses can appear in a group together, thus building a Groupwise RandomForest (GRF) classifier. The RF may then be conditioned on featurescalculated between ROIs, or, groupwise features, thus building aGroupwise Conditional Random Forest (GCRF) classifier. The quality of agiven group of ROIs may then be estimated.

Example Region of Interest Features

A notation for describing ROI features is first discussed. Each ROI maybe loaded from a DICOM RT plan file as a set of 3-dimensional pointsalong the ROI's surface, denoted by ψ with |ψ| points. There may be acorresponding DICOM CT image volume, I(x) where x∈Ω, the image domain,with |Ω| voxels. S(x) may be defined as a binary representation of theROI, with 1 for object and 0 for background, and a corresponding signeddistance function (SDF) representation ϕ(x), with positive values insidethe object and negative outside [50].

Individual ROI features may be calculated for a single ROI, as opposedto a group of ROIs. They can be grouped into two broad categories: shapeand intensity.

Shape features may involve both affine invariant and affine dependantfeatures, and it may be left to the learning stage to automaticallydetermine if affine dependant features are reliable for a given ROIclass. Various treatment protocols may demand that a patient be scannedin a consistent pose, and thus affine dependant features may be usefulfor those resulting ROIs. Examples of ROIs features are summarized inTable 1, FIGS. 3a-3b . Vector valued features, e.g. histograms, may beconcatenated with scalar features to create a single final featurevector per ROI. Though seemingly simplistic, ROIs of different classesmay have the exact same shape, and so no set of shape features may besufficient to distinguish between them. For these ROIs densityinformation may be first used, and then groupwise information.

Intensity features may capture information about the densitydistribution for a given ROI, such as listed in FIG. 3b . They mayenable distinguishing between two equally shaped objects that havedifferent densities within their boundaries, such as an esophagus andspinal canal in CT, for example.

The task of classifying a single ROI given its individual features usingRF is discussed below.

Random Forests for Region of Interest Classification

FIGS. 4a-4d show progressively descriptive example graph structures forautomatic ROI classification. Dashed lines in (d) are used to indicatethat only select features within the plates are linked to the classlabel variables, c_(*,j). Connections between all feature, class labelvariable pairs are not shown to keep the graph more easilyinterpretable.

First introduced by Leo Breiman in 2001, RF are a generalization ofdecision trees that use a mode-based voting algorithm over a set orforest of decision trees [42]. A new sample may be classified by eachdecision tree, and then the mode of the output over the entire forestmay be taken as the final output class. A detailed review of RFs isoutside the scope of this disclosure, and a discussion may be found in[51]. The following summarizes the RF approach in its application to ROIclassification and describes some details to assist in understanding thepresent disclosure.

Let P_(j) be an individual RT plan from a set of plans P. Each plancontains one or more ROIs, denoted as {C_(1,j) . . . C_(k) _(j) _(,j)}for plan P_(j) with k_(j)∈[1,∞) ROIs. Each ROI may take one of |C| classlabels where c_(i,j)∈C is the class label for ROI C_(i,j), and C is theset of possible class labels (e.g. heart, breast, lung). A set offeatures, F, may be defined with N features per ROI. An individualfeature {F_(h,i,j):h≤N} may be calculated from plan P_(j), and ROIC_(i,j). Rather than always writing 1 . . . n for some set with nelements, * has been used to denote taking the entire set over aparticular index, or group of indices. Therefore, F_(*,1,1) is the sameas {F_(1,1,1) . . . F_(N,1,1)}. Indices have also been replaced by ROInames to indicate all ROIs of that class, as a list of ROIs for a givenplan is unsorted. For example, F_(*,heart,*) is the set of all featuresfor all heart ROIs taken across all plans. Lastly, the relativecomplement of a set has been denoted by \. For example, F_(*\h,1,1) isequivalent to {F_(1,1,1) . . . F_(h−1,1,1), F_(h+1,1,1) . . .F_(N,1,1)}. For ease of reference the notation used herein is summarizedin Table 2, FIG. 5.

Given a set of features, F_(*,i,j), the goal is to predict the classlabel c_(i,j). Casting this as an inference problem, the goal is tolearn P(c_(i,j)|F_(*,i,j)), denoting the probability of ROI C_(i,j)being assigned class label c_(i,j), given the jointly observed featuresF_(*i,j). Inference may be performed to obtain the class label using amaximum-a-posteriori (MAP) estimate as:c* _(i,j)=argmax_(c) _(i,j) P(c _(i,j) |F _(*,i,j)⋅)  (1)

Various methods may be suitable to potentially learnP(c_(i,j)|F_(*,i,j)); in this example, RFs are used because theinference and learning are typically both reasonably fast [51]

RF Learning

P(c_(i,j)|F_(*,i,j)) may be estimated from M training plans,P_(1 . . . M), with known ROI class labels c_(*,1 . . . M) andcorresponding features F_(*,*,1 . . . M). The training may be abootstrapping method using cross-validation to build a strong learner,the forest, from a combination of weaker learners, the decision trees[42]. Assume the training set consists of R ROIs, and that the forestwill constitute a set of trees, T.

To build a particular tree, T_(t), a subset of ROIs, r, may be chosen atrandom with replacement from the set of R training ROIs. This subset ofROIs may form the training set of an individual tree, and the test setfor each decision tree may be all the ROIs not in the tree's individualtraining set (which may be referred to as the tree's out-of-bagsamples). Out-of-bag samples effectively may give the RF method abuilt-in form of cross-validation, where each tree is trained on some ofthe set of R ROIs and tested on the remainder. For each node in thetree, a subset of ROI features, F_(1 . . . n,r,1 . . . M), where n<<N,may be randomly chosen. Features F_(1 . . . n,r,1 . . . M) may be usedto make a classification decision at that node by calculating the bestsplit based on Shanon entropy [51].

Every branch node now may represent a binary split over a particularsubset of features. Given a particular feature sample, f=F_(*,i,j), thetree can be traversed down to a leaf node. During training, for everyclass label, the number of training samples that reach the leaf node maybe counted and used to estimate P_(t)(c_(i,j)|F_(*,i,j)=f). This may berepeated for all leaf nodes, creating an empirical estimate ofP(c_(i,j)|F_(*,i,j)), taken as the mode of P_(t)(c_(i,j)|F_(*,i,j)) overall trees in T.

RF Inference

Given a novel ROI, C_(i,j), features, f=F_(*,i,j) may be computed, andthen every tree in the forest may be traversed andP(c_(i,j)|F_(*,i,j)=f) may be computed as the mode across the forest.The assigned class label for C_(i,j) may be then calculated according to(1).

An observation is that (1) only considers a single ROI, whereas eachplan can contain a group of ROIs. The graph structure thus far isdepicted in FIGS. 4a and 4b . The joint probability may be estimated foran ROI and its observed features (FIG. 4a ), but all new ROIs in a planmay be considered to be independent (FIG. 4b ). In essence, thoughP(c_(i,j)|F_(*,i,j)) has been modelled thus far, the goal is to modelP(c_(*,j)|F_(*,*,j). The next two sections discuss how to approximatethis model and perform inference.

Groupwise Random Forests

Though RFs are an increasingly popular learning algorithm, in theirdefault incarnation they typically classify all input data independently[51]. In the context of the present disclosure, this may mean whenencountering a group of ROIs the resulting classifications may make noattempt to ensure consistency across the ROI labels. For example, ananal canal ROI is unlikely to appear in the same plan as a breast ROI,and this relationship should be accounted for during learning andinference. At this point the set of output labels c_(*,j) is of concern,and not the specific distribution P(c_(*,i,j)|F_(*,*,j)), thus it may besufficient to ensure the distribution is considered in the inference,even if not directly calculated. The model becomes a CRF [52]. Note,however, that this is not what is referred to the conditional RF, aswill become clear in the next section.

At this stage the CRF models the connections between unobserved ROIclass labels, c_(*,j). Rather than enforcing a particular degree ofconnectivity amongst the different classes, it has been learned whichclass labels are independent, and thus the graph topology. An example ofthe fully connected class labels, with corresponding conditionalfeatures, is shown in FIG. 4c . Consider the complete distribution interms of potential functions, ψ, defined over the maximal cliques in thegraph. Further below is a discussion of how to learn those potentials.

The groupwise potential function may take the form:

$\begin{matrix}{{{P( {c_{*{,j}},F_{*{,{*{,j}}}}} )} = {\frac{1}{Z}{\psi( c_{*{,j}} )}{\prod\limits_{i = 1}^{k_{j}}\;{\psi_{i}( {c_{i,j},F_{*{,i,j}}} )}}}},} & (2)\end{matrix}$

where Z is the partition function.ψ_(i)(c_(i,j),F_(*,i,j))=P(c_(i,j)|F_(*,i,j)), learned from the RFmodel, may be set. Application of the product rule is expected to leadto a conditional density of:

$\begin{matrix}{{P( {c_{*{,j}}❘F_{*{,{*{,j}}}}} )} = {\frac{1}{Z \times {P( F_{*{,{*{,j}}}} )}}{\psi( c_{*{,j}} )}{\prod\limits_{i = 1}^{k_{j}}\;{{P( {c_{i,j}❘F_{*{,i,j}}} )}.}}}} & (3)\end{matrix}$

The result is expected to be an inference problem of the form:

$\begin{matrix}{{c_{*{,j}}^{*} = {\arg\;{\max_{c_{*{,j}}}{{\psi( c_{*{,j}} )}{\prod\limits_{i = 1}^{k_{j}}{P( {c_{i,j}❘F_{*{,i,j}}} )}}}}}},} & (4)\end{matrix}$

where

$\frac{1}{Z \times {P( F_{*{,i,j}} )}}$has been removed because it may be expected to have no effect on themaximization over c_(*,j). Finally, taking the negative logarithm of thelikelihood is expected to yield the minimization problem:

$\begin{matrix}{c_{*{,j}}^{*} = {\arg\;{\max_{c_{*{,j}}}( {{- {\log( {\psi( c_{*{,j}} )}^{-} )}}{\sum\limits_{i = 1}^{k_{j}}\;{\log( {{P( {c_{i,j}❘F_{*{,i,j}}} )} \cdot} )}}} )}}} & (5)\end{matrix}$GRF Learning

In order to perform inference on (5), the class label potentialfunction: ψ(c_(*,j)) should be learned. An aim may be to learn a binarypotential function where ψ(c_(*,j))=1 for all valid class labelassignment groupings, and zero otherwise. This may be a design choice toensure that rare groupwise label configurations are not discountedsimply because only a few patients need that particular set of ROIs fortreatment. Essentially, this prior may describe all feasible group labelconfigurations as equally probable, while all other group labelconfigurations are infeasible. This choice may have the added benefit ofreducing inference difficulty, as discussed shortly. In practise a lessrestrictive prior could be learned from class label histograms but thismay increase inference difficulty and impose an undesired prior. Usingthe M training plans, P_(1 . . . M) discussed above, the set of allunique class label assignments groupings, G, may be learned, where G_(p)denotes particular grouping. The groupwise class label potentialfunction, ψ(c_(*,j)), may not in general be submodular. Exactoptimization of (5) may be expected therefore to be NP-hard [53].

GRF Approximate Inference

The inference algorithm may be based on the popular graph cuts technique[54, 55], but inference may be formulated as a generalized assignmentproblem. For the moment, the class label interaction potential functionψ(c_(*,j)) may be ignored. The cost of assigning a label to a ROI may beset as −log(P(c_(i,j)|F_(*,i,j))). The best assignment is expected tohave minimal cost (although other optimization techniques may bepossible, for example the best assignment may be expected to havemaximal cost in some cases), while ensuring that each class only appearsa set number of times. The problem can be expressed as the followinglinear program:

$\begin{matrix}{x^{*} = {\arg\;{\min_{x}{- {\sum\limits_{{u \in C}\;}{\sum\limits_{i = 1}^{k_{j}}\;{x_{u,i}{\log( {P( {c_{i,j} = {u❘F_{*{,i,j}}}} )} )}}}}}}}} & (6)\end{matrix}$subject to the constraints:

$\begin{matrix}{{{{\sum\limits_{i}^{k_{j}}\; x_{u;i}} \leq {1\mspace{14mu}{for}\mspace{14mu} u}} \in C}{{\sum\limits_{{u \in C}\;}x_{u;i}} \leq {1\mspace{14mu}{for}\mspace{14mu} i} \leq k_{j}}{{x_{u;i} \geq {0\mspace{14mu}{foru}}};{i \in C};{k_{j}.}}} & (7)\end{matrix}$

The variable, x_(u,1), may be a binary indicator variable that assignsROI class label u∈C to ROI c_(i,j). As written this may be considered tobe a standard assignment problem [56]. The first constraint may ensurethat at most one instance of a given ROI label occurs in a single plan.The second constraint, in combination with the third constraint, mayensure that a single ROI is assigned only one label. Since theconstraint matrix may be totally unimodular there may always be at leastone integral valued solution, thus there may always be an optimalsolution where x will be binary (i.e. assigning each ROI a single labelinstead of a fraction of labels). Since the program is linear, anyoptima may be expected to lie at the bounds of constraint space andthose bounds are integral. This may be relevant because enforcingx∈{0,1} may create a harder optimization problem [56].

An approximation to ψ(c_(*,j)) may be made by modifying the constraintsof (6) to:

$\begin{matrix}{{{{\sum\limits_{i}^{k_{j}}\; x_{u;i}} \leq {L_{u}\mspace{14mu}{for}\mspace{14mu} u}} \in C}{{\sum\limits_{{u \in C}\;}x_{u;i}} = {{1\mspace{14mu}{for}\mspace{14mu} i} \leq k_{j}}}{{x_{u;i} \geq {0\mspace{14mu}{foru}}};{i \in C};{k_{j}.}}} & (8)\end{matrix}$

This modification may ensure that at most L_(u) instances of a given ROIclass occur in a single plan. For example, a plan can only have oneheart ROI, but can have multiple targets. L_(u) can be readily computedby taking the maximum number of times a given ROI appears in thetraining set. This may be also a design choice, and in practise L_(u)could be set to ∞ if no constraint is desired. Note, however, that thismodification may partially model ψ(c_(*,j)) as it allows a heart and alung, but not two hearts, for example. In essence this may model thegroupwise potential function between all instances of the same classlabel. The groupwise potential between different classes is considered.

ψ(c_(*,j)) may be introduced by creating a quadratic program over thebinary indicator variables, but the result may be a non-convex quadraticwhenever ψ(c_(*,j)) is not submodular. The non-convexity may stem fromtrying to restrict groupings with non-empty intersects, such as tryingto enforce an XOR relationship like {lung, heart, breast} and {lung,heart, esophagus}, but not {lung, heart, breast, esophagus}. Suppose,however, that it is known that a particular set of ROIs c_(*,j)contained labels from G_(p). The following uninodular constraint may beadded to (6):

$\begin{matrix}{{{\sum\limits_{u \notin G_{p}}x_{u;i}} = {{0{fori}} \leq k_{j}}};} & (9)\end{matrix}$

which may ensure that the assigned class labels will be in G_(p). Thismay lead to a natural, though approximate, inference algorithm for (5).

First, the most likely grouping may be determined as:

$\begin{matrix}{G_{p}^{*} = {\arg\;{\max_{G_{p}}{\sum\limits_{{u \in G_{p}}\;}{\sum\limits_{i = 1}^{k_{j}}{\log( {{P( {c_{i,j} = {u❘F_{*{,i,j}}}} )},} )}}}}}} & (10)\end{matrix}$

which may be the expected group given the observed features. Then (6)may be optimized with added constraints (8) and (9).

GRF Exact Inference

Though exact inference of (5) may be expected to be NP-hard, takingadvantage of the linear relaxation in combination with the problemstructure can provide reasonable run times. Notice that −log(ψ(c_(*,j))in (5) may be zero for any viable group configuration, and ∞ for allothers. Any grouping outside of G may have infinite cost, and so onlyclass label groupings in G may be optima. Provided there are a smallnumber of feasible groups, one may need only optimize (6) with addedconstraints (8) and (9) for each group G_(p)∈G. The result may be an x*for every group G_(p)∈G, and the one with the minimal objective valueaccording to (6) may be the global optima of (5). In practise therecould be an exponential number of groups in G, and so no theoreticalbound on running time may be broken.

Building from the previous section, the example model may now considerjoint-wise potentials over class labels enabling a groupwise assignmentof labels given the RF generated posterior distributions over features.However, consider the following example feature relationship: lungs arelarger than hearts. Currently the model may have no way to learn thisrelationship, or to use it during inference. A challenge duringinference may be that which ROI is the lung may be unknown, andtherefore the feature itself may be a latent variable. The followingsection discusses how to calculate groupwise features, condition the RFson these variables, and how to use the groupwise features duringinference.

Groupwise Features and Groupwise Conditional Random Forests

Groupwise features may be defined as those calculated between groups oftwo or more ROIs. Although this example focuses on features calculatedbetween pairs of ROIs, in practise larger groupings could be used.Examples of groupwise features are listed in Table 3, FIG. 6. Thesefeatures can be calculated between pairs of ROIs, but a question may bewhich pairs of ROIs to calculate a feature for. So, for example,consider the distance between ROIs. When performing inference on a novelROI the inference method may need to know if the pairwise distance isbetween the novel ROI and a lung or the novel ROI and an anal canal, forexample. For example, a small distance to the lung ROI may indicate apossible heart label, while the anal canal may preclude the heart label.Essentially, both a groupwise feature and the class it is calculatedwith respect to may need to be known for inference. While being close to“some” other ROI may be a valid feature, it may not be nearly asdiscriminative as being close to an ROI with a known class label. Ingeneral, pairwise groupwise features can be written asg(c_(b,j),F_(*,a,j),F_(*,b,j)) for ROI C_(a,j) calculated with respectto ROI C_(b,j). Groupwise features may be restricted to at most oneunobserved ROI label, but multiple observed ROI labels can be used.Discussion of this appear in [48].

The remainder of this section discusses how to perform learning andinference on the graph in FIG. 4d , while also learning the structure ofsaid graph (i.e. between which pairs of ROIs to build the features). Aswith the previous section, the complete distribution is first consideredand then inference is discussed.

The groupwise potential function with groupwise features may take theform of:

$\begin{matrix}{{P( {c_{*{,j}},F_{*{,{*{,j}}}}} )} = {\frac{1}{Z}{\psi( c_{*{,j}} )}{{\psi( {c_{*{,j}},\; F_{*{,{*{,j}}}}} )}.}}} & (11)\end{matrix}$

where ψ(c_(*,j)) is discussed above, and ψ(c_(*,j),F_(*,i,j)) may bewhere g(c_(*,j),F_(*,*,j)) is incorporated. The potential function,(11), may include the seemingly non-maximal clique ψ(c_(*,j)) to handlethe graph pruning that may occur during learning (indicated by themissing dotted connections in FIG. 4d ). Consider the conditionaldistribution assuming all but the ith ROI class label:P(c _(i,j) |c _(*\i,j) ,F _(*,*,j)),  (12)where c_(*\i,j) denotes having observed all ROI class labels for planP_(j) except for the ith. This distribution can be learned by RFs,similar to how P(c_(i,j)|F_(*,i,j)) was learned in the abovediscussions.GCRF Learning

During training the labels for all ROIs may be known, and thus thegroupwise features can be calculated, and (12) can be learned. Achallenge may be to learn which features can be reliably calculatedduring inference on novel data. A groupwise feature can be reliablycalculated if its dependent class labels can be determined withoutgroupwise features. For example, suppose P(c_(1,j)=lung|c_(2 . . . k)_(j) _(,j),F_(*,*,j))≈P(c_(1,j)=lung|F_(*,1,j)), then observing a lungROI may be independent on all groupwise features and all other ROIlabels. Thus it may be possible to infer if c_(1,j)=lung with highaccuracy using only F_(*,1,j). How well this condition holds may bemeasured by checking how well (1) predicts a given class for out-of-bagsamples, also known as the out-of-bag-error [42]. As this may only makeuse of ROIs from the training plans, P_(1 . . . M), it may beeffectively performing cross-validation of the RF over different randomsubsets of the training data.P(c_(i,j)κ_(*\i,j),F_(*,*,j))≈P(c_(1,j)|F_(*,1,j)) may be assumed forall classes with out-of-bag error below a threshold. A threshold errorrate of 5% may be used, for example. By learning whereP(c_(i,j)|c_(*\i,j),F_(*,*,j))≈P(c_(i,j)|F_(*,i,j)), the graphconnectivity in FIG. 4d may be learned. The learned graph may only havegroupwise feature connections for ROI classes for whichP(c_(i,j)|c_(*\i,j),F_(*,*,j))=P(c_(i,j)|F_(*,i,j)) does not hold. Theresulting set of ROI classes, which may be referred to as stableclasses, are denoted by τ. In order to learn (12) via RF the stablegroupwise features may be substituted as:P(c _(i,j) |c _(*\i,j) ,F _(*,*,j))=P(c _(i,j) |F _(*,i,j) ,g(c _(τ\i,j),F _(*,i,j) ,F _(*,τ\i,j))),  (13)

which may assume that c_(i,j) is independent from all class labels notin T, and that groupwise features involve at most one non-stable ROI.The groupwise relationship between class labels not in T may be modelledas discussed above by ψ(c_(*,j)). Modelling groupwise features betweennon-stable ROIs is not discussed in detail here, but may be carried outusing suitable methods. Using the product rule, assuming all stable ROIsare observed, and substituting in (13), (11) may be approximated as:

$\begin{matrix}{{{P( {{c_{*_{{\backslash T};j}}❘c_{T;j}};F_{{{*;}*};j}} )} = {\frac{1}{Z}{\psi( c_{*{;j}} )}{\prod\limits_{{i = 1};}^{k_{j}}\;{{P( {{C_{i;j}❘F_{*{;i;j}}};{g( {c_{{T\backslash i};j};F_{*{;i;j}};F_{*_{;{T\backslash i};j}}} )}} )}.}}}}\;} & (14)\end{matrix}$GCRF Approximate Inference

An approximate inference may be based on the learned independencerelationships involving the stable classes. The stable class labels maybe estimated using (1), and then the groupwise features may becalculated. Inference can then be performed using approximate or exactmethods, such as discussed above, but substituting inP(c_(i,j)|F_(*,i,j),g(c_(τ\i,j),F_(*,i,j),F_(*,τ\i,j))) forP(c_(i,j)|F_(*,i,j)).

Table 4a, FIG. 7a , provides a summary comparing classification accuracyresults for various example RF algorithms.

Table 4b, FIG. 7b , provides a summary comparing classification accuracyresults based on a more comprehensive set of example data.

Inferring Contour Quality

Using the class posterior distributions [56], contour quality may beestimated, under the assumption that the contouring errors may beexpected to represent a larger degree of feature variation than thatintroduced by anatomical variability. That is to say, a heart may beexpected to generally look like a heart, but a mis-contoured heart maybe expected to look far less like a heart than any natural heart would,and hence may be expected to have a low class posterior given itsfeatures. For example, a jagged heart ROI may be anatomicallyimplausible and so its features may not strongly predict the heartclass. P(c_(i,j)|F_(*,i,j)) may be used to directly estimate contourquality for the stable ROIs, sinceP(c_(i,j)|c_(*\i,j),F_(*,*,j))≈P(c_(i,j)|F_(*,i,j)) for these classes.For the non-stable ROIs,P(c_(i,j)|F_(*,i,j),g(c_(τ\i,j),F_(*,i,j),F_(*,τ\i,j))) may be used.Whether using stable or non-stable ROIs, the automatic quality estimatemay be defined as ε, scored between 0 and 1.

Example Plan Classification Algorithm

An example plan classifier may be built in a similar manner as the ROIclassifier discussed above. Various suitable classification algorithms(e.g. Randon Forests, Support Vector Machines) may be used that do notassume independence between input variables, as opposed to other methodslike Naïve Bayes. A challenge in plan classification may be that eachplan can have 1 to many fractions, each fraction can have 1 to manybeams, and each beam can have 1 to many control points. Theserelationships may need to be built into the classifier.

An example approach to deal with this may be to use dictionary learning[57-60], and histograms. Unsupervised learning methods (e.g. K-means,sparse coding, sparse autoencoders, etc., as discussed in [46, 47,57-59, 61]) may be used to learn a dictionary for control points. Novelcontrol points may be assigned to their most similar dictionary codeword, and a histogram may be computed over all control points for eachbeam. A beam dictionary may be then learned using individual beamfeatures along with the control point histogram for the respective beam.Similarly, a dictionary for fractions may be learned using a histogramover beams. Finally, using a histogram over fraction code words, inconjunction with other plan features, a plan classifier can be trained.In the example of a probabilistic classifier, this approach may modelthe probability of observing a particular plan class given adistribution of fractions, where fractions may be recognized based onfraction features and distributions of beams, and beams may berecognized based on beam features and distributions of control points.This method may recognize a plan based on its configuration of controlpoints per beam, creating a configuration of beams per fraction, andfinally a configuration of fractions. Example results for this exampleembodiment are discussed below.

A related approach may be to use deep learning structures like [62-65],with shared weights for all of the control points in a beam, beams in afraction, and fractions in a plan.

A separate approach may be to consider fractions, beams and controlpoints as independent features. A dictionary may be built for controlpoints as discussed above. A dictionary may be then built for beams, butwithout including the control point histogram as an input feature. Adictionary may be also built for fractions, but without considering thebeam histogram. Finally, the fraction, beam, and control point code wordhistograms for each plan may be used as features. In contrast to theabove example, in a probabilistic classifier this example may estimatethe probability of a class label given plan features in conjunction witha distribution of fractions, a distribution of beams, and a distributionof control points. This method may not model that a particular beamoccurred within a particular fraction, for example.

Another example approach may be to consider individual beams, fractions,and control points independently, rather than modelling jointdistributions.

Regardless of the particular embodiment, a set of plan features may becomputed for each plan, and may be used to build a classifier andperform quality estimation (via density estimation, or regression, etc.)in addition to the patient features.

Patient features can be taken directly from a patient's chart (e.g.age), or from the acquired CT image and associated plan dose map.Patient features can be as rudimentary as a histogram over imageintensity, or more advanced patch-based image features with dictionarylearning [57-60, 66], of any other such feature. An aim may be tocalculate features that may ensure similar treatment sites have similarfeatures, and that patients with similar geometry have similar features.When those features are combined with plan and dose-map features theymay ensure that the new patient is receiving treatment in-line with howsimilar historical patients (e.g., from an organ geometry and appearancestand-point) have been treated in the past.

Another approach may be that of OVH [33], and other approaches in thefield of content-based image retrieval. Using a new patient's geometricfeatures (e.g., organ geometry and appearance) to find similar patientsin a database of historical treatments may ensure that the new patientis treated with a plan in-line with his/her most similar historicalcounterparts.

Example of Automated Segmentation Algorithm

Calculating patient geometry features may be made easier with labelledimage data (e.g., segmentations or ROIs). Various suitable automatedsegmentation techniques (such as discussed in [67, 68]) may be used tosegment additional image structures automatically and enable thecalculation of additional features relating to how much dose certainstructures are receiving, or the geometry of the structures (e.g.features used for ROI classification algorithm). Just as with the ROIsin the plan already, these ROIs can be processed through the ROIclassifier and quality estimates algorithms to help ensure they are ofsufficient quality prior to plan classification and quality estimation.

Example of Plan Error Detection

An example plan error detection algorithm can be designed in a similarmanner as the ROI error detection algorithm, directly measuring qualityfrom the probabilistic classifier output as in the case with RandomForest results used for ROI quality estimation.

Alternatively, using density estimation algorithms may provide a directquality estimate in the form of a probability of observing a particulartreatment plan given a combination of patient and treatment features. Anexample of this algorithm was used for the GUProstateVMAT results,described further below.

When both accepted and rejected treatment plans are available ashistorical data for training, then a classifier or regression algorithm(e.g., using a setting of 0 for acceptable plans, and 1 for rejectedplans) such as Random Forest can be used to directly classify treatmentplans and/or output a quality estimate (e.g., using Random Forestregression). An example of this algorithm was used for theBreastLeftCavityIMRT results, described further below.

Example of Integrated Classification Algorithm

Integrating ROIs and plans can be performed in any of the previousexamples. A plan may have zero to many ROIs. Using the ROI classifier,all ROI labels may be known, and a distribution of ROIs can becalculated as a plan feature. This may model that a chest plan shouldtypically include a left and right lung ROI, for example. The ROIclassifier may already ensure that the heart ROI is smaller than thelung ROI, for example. Running the ROI classifier prior to integrationmay allow this phase to include specialized ROI based features, forexample, the dose to the heart specifically. This feature may not becalculated prior to knowing which ROI is the heart. This method may thusprovide a more natural workflow. A planner may first create ROIs, andthen may check their quality using the ROI classifier. Once the ROIs areapproved the planner may generate a plan. The plan classifier may thenuse ROI-based features as additional information to determine planquality (e.g., dose to ROI, ROI shape, etc.). An aim of this may be toensure that similar patients result in similar patient features. Thequality estimation algorithm may then check how well the patientfeatures pair up with the plan features, based on whether similarhistorical patients received historical treatment plans planned in asimilar manner.

Another suitable method may be to build an integrated classifier thatmay use plan features to assist with ROI classification, and vice versa.

Various other approaches to detecting errors in ROIs and plans may alsobe suitable. An example approach may model any set of features rarelyobserved as a potential error using the learned probabilities in theclassifier or a density estimation technique P(F|class). For example,this approach is used in the ROI classifier detailed in TMI [48].Another example approach may be to expressly train a classifier orregression algorithm on erroneous ROIs and plans. These other approachescan be used separately or together for greater flexibility.

Example of Quantitative Feature Analysis

Quantitative feature analysis may include estimating which features of aplan are erroneous, and performing any manual boundary checks on theplan (for example ensuring that the dose is less than a maximumhard-coded safe level for a particular patient age group). Thesefeatures and rules can be taken directly from established medicaltreatment guidelines, for example.

The method for estimating which feature is responsible for a low planquality may be dependent on which specific quality estimation algorithmis used. For example, with density estimation techniques, Bayes rule canbe applied to estimate the probability of observing a particular planfeature (e.g. number of beams) given all other features. Or theprobability of observing a plan feature (e.g. number of beams) can becalculated assuming patient features are observed, and summing out overall possible values for other plan features. This may, for example,report that 10 beams are rarely if ever used for these types ofpatients, and the planner should instead use 6.

Example of DICOM-RT Data Verification

An example of data verification may be paired with the Random Forestclassifier (e.g., as discussed above) to help ensure that both planversions end up at the same leaf of the trees. An alternative approachmay be to ensure that the feature vectors match, and flag any step ofthe plan production pipeline that accidently change a feature.

Example Automated Dose Inference Algorithm for Automated TreatmentPlanning

In some examples, the automated treatment planning may build uponstate-of-the-art methods from machine-learning and image processing.Image features, i.e. radiomic data, may be computed from a patient CTimaging and then mathematical regression may be used to infer what thepersonalized RT dose map should look like for that patient. The basictechnological premise is that a given patient whose treatment is beingplanned would be expected to have a similar treatment plan to anexisting similar patient in the historical database of treated patients.However, the disclosed method operates on a finer-grain level thansimple patient-to-patient matching and learns more specific detailsrelating the dose and anatomy. For example, the algorithm can learn thatthe corner of a lung with particular appearance (size, density) shouldbe irradiated a particular way. In essence, the inference may match anovel image patch representing a 1×1×1 cm cubic region of the patient tomillions of other patches in the database and then use the dose levelfrom the most similar patches.

Radiomic features may be used to describe the image patches, and machinelearning may be used to learn how to judge similarity between patches ina way that will accurately predict dose. In this way, it may be possibleto recognize the difference between regions to avoid and target regionswithout requiring the standard approach of diligently delineating ROIsof the anatomy beforehand. For example, in breast RT, regions of theimage recognized as the heart should typically be avoided and theproposed dose for these regions will be low, but regions at theinterface between the lung and the breast and typically should beirradiated with a predicted dose close to the prescribed dose.

Inference of the proposed dose map may include the steps: i) Access amass of exemplar historical data in the form of clinical treatment planswith corresponding CT images, and optionally divide the set into atraining set and independent testing set (in some examples, anindependent testing set may not be necessary); ii) Extract meaningfulimage features, i.e. radiomic data, from the CT data (e.g. texture,local image appearance, gradient); iii) Use a regression algorithm (e.g.Decision Forests [42]) to learn a non-linear multivariate regressionmodel from the data that is capable of predicting the dose for novel CTimages; and iv) Optionally, validate the predicted dose using theclinically delivered treatment plans for the testing data.

Example Automated Dose Inference Algorithm for Adaptive RT

The methodology described in the previous section may be expended toinclude information about previous treatment plans for use in apersonalized adaptive RT context. The disclosed treatment planningmethods may use imaging acquired during the course of treatment to adaptto anatomical changes that may result from treatment.

In some examples, the present disclosure may provide an automated,personalized adaptive RT method that involves training a suitableregression model [41, 42, 46] for the specific patient after the initialtreatment planning has been approved, and that incorporate the delivereddose from each treatment thereafter. When changes in anatomy occur(e.g., rigid and/or deformable changes), these may be detected based onimaging acquired for each treatment day. The predicted dose may then beautomatically updated for the current imaging data by determining thecorresponding image patches in the context of the training databaseaugmented with the imaging and the dose map from the previous treatmentplans for that patient.

Adaptive RT may include the steps: i) adding the patient's treatmentplan and corresponding repeat imaging to the historical database tore-train the machine learning algorithm to incorporate time-specificdata to tailor to this specific patient; ii) calculate newtime-dependent features based on repeat patient imaging; iii) infer anupdated dose to account for anatomical changes; and iv) optionallyvalidate for each imaging time point.

Example Validation and Results

The examples below discuss validation of an example embodiment of boththe ROI and plan classification algorithms for an example automated QAmethod.

Example ROI QA Results

In this example, data consists of 17,579 ROIs from 1574 deliveredtreatment plans with 77 ROI classes. The data was gathered over one yearfrom the Princess Margaret Cancer Centre in Toronto, Ontario. Each plan,created by a treatment planner (dosimetrist), contained a set of expertlabelled ROIs, and had a corresponding DICOM CT image. ROIs may haveresulted from manual or semi-automatic segmentation, but not fullyautomatic segmentation. Every plan was reviewed by multiple expertsbefore being used for treatment in accordance with health and safetystandards for RT. There are expected to be errors in the data. Some ROIshave been incorrectly labelled, and others have been incorrectly drawn.The example disclosed method was found to learn in-spite of theseerrors, and detect these errors.

Contour mislabelling errors may be directly detected via automated ROIclassification. If the automatic ROI class label disagreed with the ROIclass label in the input treatment plan, a mislabelling error may bereported. Contour quality assessment may be based on the posteriorprobabilities, as outlined above. This example study includes two typesof experiments, those dealing with automatic ROI classification, andthose relating to automatic quality assessment.

In order to evaluate the automatic quality assessment, a subset of 303ROIs from 41 ROI classes was specifically re-evaluated by an expert tocheck for contouring errors. Only ROI classes for which theout-of-bag-error was lower than 10% were used.

Automatic Contour Classification

All classification experiments were performed using repeated randomsub-sampling validation. The data was randomly split into two disjointclasses, with 941 (60%) for training plans and 639 (40%) testing plans.All learning and parameter tuning was done using the training data only,and then accuracy was evaluated on the testing set. Parameters includedthe maximum depth of each tree, and the number of variables examined persplitting node, n (e.g., as discussed above). The validation process wasrepeated for 40 random sub-samplings and the reported results wereaveraged across the random splits. Note that because plans and not ROIswere split into training and testing groups, the resulting ROI trainingand testing percentages were different.

Parameter sensitivity of the example method studied is depicted in FIGS.8a-8d , and learned feature importance is shown in FIGS. 9a -9 b.

FIGS. 8a-8d illustrate parameter sensitivity in the example RF (8 a, 8c) and conditional RF models (8 b, 8 d). Error bars are presented acrossthe 40 random sub-samplings of training and testing data. Out-of-bagaccuracy is presented in blue (indicated as “1”), with accuracy for thetesting data in red (indicated as “2”). The example models were found tobe stable over a relatively wide range of parameters and did not exhibitany strong evidence of over-fitting.

FIGS. 9a-9b illustrate feature importance learned in the example RF (9a) and conditional RF models (9 b). The groupwise features in (9 b)indicate a relatively strong influence. Vector valued features appear asa summation over the entire feature vector.

Maximal tree depth and the number of splitting variables were learnedautomatically for each of the 40 random sub-samplings by finding wherethe out-of-bag-error flattens out as a function of the dependentparameter. For this example RF the average learned maximal depth was 17,and the number of splitting variables was 37. For this exampleconditional RF, the maximal depth was 16, and the number of splittingvariables was 43. The intensity histogram contained 160 bins, and theSDF histogram contained 30. These values were set manually by examiningrandom ROIs and then fixed for all experiments.

This example learning method was compared with the canonical RFs, naiveBayes (NB), artificial neural network (ANN), support vector machine(SVM) classification algorithms. Each competing method's parameters wereestimated via cross-validation. NB and ANN implementations used built-inMATLAB toolboxes, and libSVM was used for SVM with a radial basis kernelfunction [69]. The example learning method was considered both with andwithout groupwise conditional features and groupwise inference. Whereapplicable results for the example learning method were reported usingboth approximate and exact inference. The complete set of organs at riskand other reference structures considered in this example study islisted in Table 6 (FIGS. 14a-14c ), while clinical targets are listed inTable 7 (FIGS. 15a-15c ). In cases where ROI classes (e.g., Heart)appeared in multiple treatment plan classes, only one treatment plantype has been listed. Average training and testing set sizes for eachclass are presented.

Example results are listed in Table 4a (FIG. 7a ) and Table 4b (FIG. 7b). The total accuracy of a classifier was defined as the percentage ofcorrect classifications across all classes (i.e., total number ofcorrect classifications divided by total number of testing ROIs), incomparison to the ROI labels provided by the treatment planners(dosimetrist). Total accuracy across all classes is presented, for OARs,and for targets. Results were averaged across the 40 randomsub-samplings. The “Accuracy Std. Dev.” is the standard deviation of thetotal accuracy across all 40 random sub-samplings of the training andtesting data. The true positive and false positive classification rateswere calculated for each ROI class. True positive rates (TPR) and falsepositive rates (FPR) are presented for each individual ROI class inTable 6 and Table 7 (FIGS. 14a-14c and 15a-15c ) for the exampleGroupwise Conditional RF method. Statistics for TPR and FPR in Table 4a(FIG. 7a ) were computed across all ROI classes unless otherwise noted.The “Avg. TPR” was calculated for each individual ROI class as anaverage over the 40 random sub-samplings, and then the average TPR wascalculated across all 77 classes. These measures may indicate theaccuracy of the system in data mining applications, or in fullyautomatically re-labelling of ROIs.

The results for flagging potential labelling errors for expert reviewmay be related, but somewhat different. The overall TPR for detectingROI mislabelling errors across all classes was lower bounded by thetotal accuracy of the classifier, 91.58%. The FPR for mislabellingerrors was one minus the total accuracy, or 8.42%. The system was foundto correctly identify any ROI mislabelling error for which it correctlyclassifies the ROI, and may miss any mislabelling for which itincorrectly classifies the ROI. The TPR was lower bounded by the totalaccuracy because the example classifier flagged an error wherever itdisagreed with the input plan, even if it is also wrong about the ROIlabel in question. When the classifier outputted the same incorrectlabel as the input plan, a ROI mislabelling may go un-noticed. In otherwords, a mislabelling error false negative for a given ROI class mayonly occur when a classifier false positive is of the same class. Leftlung had a mislabelling error detection TPR of 99.96%, or one minus theFPR from Table 6 (FIGS. 14a-14c ). The average TPR across all ROIclasses for mislabelling error detection was, therefore, one minus the“Avg. FPR” from Table 4a (FIG. 7a ), or 99.89%. The expected value ofthe TPR for mislabelling error detection across all ROI classes givenobserved class distributions was 99.65%. An example of a mislabellingerror may be where the left lung has been labelled as the right, andvice versa. The example disclosed method was able to correctly identifythe mistake and accurately relabelled each lung.

Using the same analysis, a more comprehensive dataset was used in Table4b (FIG. 7b ). The dataset consisted of 59,400 ROIs from 6,199 deliveredplans and compromised 307 distinct ROI classes. The overall TPR fordetecting ROI mislabelling errors across all 307 classes was lowerbounded by the total accuracy of the classifier, 89.85%.

Automatic Quality Assessment

Whereas the previous section detailed example results for detectingmislabelled ROIs, this section deals with detecting errors relating tocontouring quality itself. A poorly drawn heart ROI is an example of anactual contouring error that, while missed during rigorous treatmentplan review, may be automatically detected by the example disclosedmethod. During treatment plan review it may be impractical for a medicalexpert to manually review every slice of every contour, and so errorslike this can often go un-noticed.

For these example experiments training was using all plans except thosecontaining one or more of the 303 manually re-evaluated ROIs. Of the 303ROIs, 48 had contouring errors, as determined by an expert. For all 303ROIs it was predicted whether or not a contouring error has occurred bythresholding the class posterior ε. The ROC curve for differentthresholds, using an example of the disclosed method, is shown in FIG.10, using bootstrapping and averaging over the FPR to compute a 95%confidence band. The area under the curve (AUC) was 0.75 with a 95%confidence interval of [0.67, 0.82] with bootstrapping or [0.63, 0.87]using another standard method [70]. Based on the ROC curve a thresholdof 63% was selected for illustrating example true and false positives.

FIGS. 11a-11c show an example false positive for lung ROIs in thepresence of a large tumour. The size of the left lung, in orange(indicated as “2”), has been reduced by the large tumour in yellow(indicated as “3”). The automatic quality estimate for the left lung isε=0.44 in comparison to the right lung (indicated as “1”) with ε=0.88.

FIGS. 12a-12c show an example false negative in a poorly contoured heartROI. The heart ROI in green (indicated as “1”) was mistakenly drawn toinclude part of the aorta. Red lines (indicated as “2”) in FIG. 12b andFIG. 12c show approximately where the contour should have ended. Theautomatic quality estimate for this heart ROI was ε=0.72. In comparison,an example jagged heart contour was found to have ε=0.54, indicatingthat the type of contouring abnormality shown in FIGS. 12a-12c may bemore common in the historical data.

Example Plan Classification Results

For validation of the example plan classification method, collected 7933clinical treatment plans were collected from 103 different treatmentplan classes over two years. The accuracy using the example embodimentdiscussed above, RF with histograms of fraction, beam, and control pointfeatures over dictionaries, was found to be 78.01% (See Table 5, FIGS.13a-13c ).

Example Automated Treatment Plan Error Detection Results

For these example experiments training was done for a single treatmentplan class at a time. In FIG. 16a the treatment plan class wasBreastLeftCavityIMRT. On a quality estimate threshold of 0.45,leave-one-out, validation was done on 86 breast plans (68 clinicallyacceptable plans and 18 rejected plans with errors, 10 of which weresimulated errors). The TPR and FPR were 0.9444 and 0.1029, respectively.The ROC curve for different thresholds, using an example of thedisclosed method, is shown in FIG. 16a , using bootstrapping andaveraging over the FPR to compute a 95% confidence band. Based on theROC curve a threshold of 45% was selected for illustrating example trueand false positives.

In FIG. 16b the treatment plan class was GUProstateVMAT. Visualizing thequality estimation space of prostate plans using multidimensionalscaling [71] to reduce the feature space to two dimensions, x1, and x2,followed by kernel density estimation [46]. The data shows 273 treatmentplans for training (black circles) and 61 treatment plans for testing(white circles). For this plan class there were 4 clinical errors (greycircles highlighted by errors).

Example Automated Dose Inference Results

FIGS. 22a-22c show example CT images. FIG. 22a shows example CT imagesfor a typical BreastLeftBreastTangent (Top) and GUProstateVMAT (Bottom)RT plan. Also shown are the corresponding automated inferred dose maps(FIG. 22b ) and clinical dose maps (FIG. 22c ) over the entire image.The dose scale shows fraction of the prescription dose. For bothautomated plans, the dose map predicted was based on images only and didnot include any delineated ROIs.

Table 10, FIG. 23 shows a summary of results from the example treatmentplan classes of FIGS. 22a-22c showing validation of dose map inferencefor automated treatment planning in 20 novel patients of each class. Thecorrelation between predicted and clinical dose maps is shown for theBreastLeftBreastTangent and GUProstateVMAT treatment plan classes.

Possible Variations and Applications

Some example variations and applications of the present disclosure arediscussed below for the purpose of illustration. Other variations andapplications may be possible.

The present disclosure may be useful for automated treatment planningand/or data mining applications, among others, as discussed below.

The present disclosure can be used for education and training oftrainees i.e. radiation oncology residents, radiation oncology fellows,radiation physics residents and RT technology students, and for RT staffi.e. radiation oncologists, radiation (medical) physicists and RTtechnologists. For example, the disclosed systems and methods can beused such that a user may perform any task during treatment planning(e.g., target and organ delineation, generating treatment plans, etc.)and may receive as output a report on the acceptability of theindividual volume delineations or treatment plans compared withhistorical plans. In some examples, this may be provided as remotelearning in which users can receive feedback without having to sit downwith a particular expert. The example system can use a remote databaseand may be system independent.

Using the same historical data that the example system is built on to doQA verification, the example system can also provide an optimizedtreatment plan and regions of interest for a particular plan class. Theexample system may produce the treatment plan class based on theparticular features from the plan and compare the new plan with theexpected plan class to see if it matches. This could also apply to ROIs.Thus, the example system can be used for image segmentation.

The present disclosure may be suited to the clinical trial environment.Examples of the disclosed methods and systems may accept data fromremote sources, interrogate the data and provide an automated analysis.This may help to ensure clinical trial compliance and provideparticipating institutions rapid review of their submission tofacilitate real-time review.

There may be many options for incorporating the disclosed methods andsystems into a clinical workflow. For example, the clinician may use anexample of the disclosed methods and systems to verify ROIs manually,semi-automatically or automatically generated before treatment planning;the planner may use an example of the disclosed methods and systems toverify and provide a pre-QA of an entire treatment plan before it goesfor review to oncology and physics; or the physicist and oncologist mayuse an example of the disclosed methods and systems to perform QA andapprove/reject completed treatment plans generated from a planner. Theseexample options may be non-exclusive. All options can be used for a planreview.

Data-mining may be another application of this disclosure. Clinicaldatabases are often fraught with errors such as mislabelled ROIs, andun-classified data. However, it may be desirable to retrospectivelysearch a clinical database and find all of the lung ROIs, or all of thelung treatment plans, or ever lung treatment plan with a heart ROI, forexample. A challenge in doing so is that databases often containincorrect or inconsistent labels, for example the left lung ROIs mightbe labelled “Left Lung”, “L Lung”, “Lung Left”, “LLung”, or evenlabelled “heart” by mistake. The present disclosure may enable users toautomatically search a database for all items of a particular class,regardless of the label, and thus find all lungs, for example,regardless of the used nomenclature. Once processed by the examplesystem, the data can be used for other research studies.

Another possible application may be the ability to use the disclosedautomated QA system to triage or flag plans that have been scored poorlyand then focus the review to these plans. As evidence by a recent pollof academic institutions [11], the implementation and conduct of peerreview rounds may be inconsistent across academic institutions withrelatively little time spent reviewing individual patients. Patienthistory, patient chart documentation, and dose prescription may be peerreviewed whereas dosimetric details related to the treatment plansincluding target coverage and normal tissue doses and technicalparameters related to the treatment plans may be typically reviewed inonly half of the cases. Therefore, an example of the disclosed QA systemmay provide a better use of time by allowing the team to focus onpotential problems flagged by the example system.

Although the present disclosure describes methods and processes withsteps in a certain order, one or more steps of the methods and processesmay be omitted or altered as appropriate. One or more steps may takeplace in an order other than that in which they are described, asappropriate.

While the present disclosure is described, at least in part, in terms ofmethods, a person of ordinary skill in the art will understand that thepresent disclosure is also directed to the various components forperforming at least some of the aspects and features of the describedmethods, be it by way of hardware components, software or anycombination of the two, or in any other manner. Moreover, the presentdisclosure is also directed to a pre-recorded storage device or othersimilar non-transient computer readable medium including programinstructions stored thereon for performing the methods described herein,including DVDs, CDs, volatile or non-volatile memories, or other storagemedia, for example.

The present disclosure may be embodied in other specific forms withoutdeparting from the subject matter of the claims. The described exampleembodiments are to be considered in all respects as being onlyillustrative and not restrictive. Selected features from one or more ofthe above-described embodiments may be combined to create alternativeembodiments not explicitly described, features suitable for suchcombinations being understood within the scope of this disclosure.

All values and sub-ranges within disclosed ranges are also disclosed.Also, while the systems, devices and processes disclosed and shownherein may comprise a specific number of elements/components, thesystems, devices and assemblies could be modified to include additionalor fewer of such elements/components. For example, while any of theelements/components disclosed may be referenced as being singular, theembodiments disclosed herein could be modified to include a plurality ofsuch elements/components. The subject matter described herein intends tocover and embrace all suitable changes in technology. All referencesmentioned are hereby incorporated by reference in their entirety.

REFERENCES

-   [1] Canadian Cancer Society|Statistics Canada. Canadian Cancer    Statistics. 2013.-   [2] Cancer Quality Council of Ontario. Radiation Treatment    Utilization. Canadian Quality Council of Ontario; 2013.-   [3] Schilling E G, Neubauer D V. Acceptance sampling in quality    control. 2nd ed. CRC Press; 2009.-   [4] Bissonnette J-P, Medlam G. Trend analysis of radiation therapy    incidents over seven years. Radiother Oncol 2010; 96:139-44.-   [5] Huang G, Medlam G, Lee J, Billingsley S, Bissonnette J P,    Ringash J, et al. Error in the delivery of radiation therapy:    results of a quality assurance review. Int J Radiat Oncol Biol Phys    2005; 61:1590-5.-   [6] Peters L J, O'Sullivan B, Giralt J, Fitzgerald T J, Trotti A,    Bernier J, et al. Critical impact of radiotherapy protocol    compliance and quality in the treatment of advanced head and neck    cancer: results from TROG 02.02. J Clin Oncol 2010; 28:2996-3001.-   [7] Abrams R A, Winter K A, Regine W F, Safran H, Hoffman J P,    Lustig R, et al. Failure to adhere to protocol specified radiation    therapy guidelines was associated with decreased survival in RTOG    9704—a phase III trial of adjuvant chemotherapy and    chemoradiotherapy for patients with resected adenocarcinoma of the    pancreas. Int J Radiat Oncol Biol Phys 2012; 82:809-16.-   [8] Purdy J a. Quality assurance issues in conducting    multi-institutional advanced technology clinical trials. Int J    Radiat Oncol Biol Phys 2008; 71:S66-70.-   [9] Ellerbroek N A, Brenner M, Hulick P, Cushing T. Practice    accreditation for radiation oncology: quality is reality. J Am Coll    Radiol 2006; 3:787-92.-   [10] Hulick P R, Ascoli F a. Quality assurance in radiation    oncology. J Am Coll Radiol 2005; 2:613-6.-   [11] Lawrence Y R, Whiton M a, Symon Z, Wuthrick E J, Doyle L,    Harrison A S, et al. Quality Assurance Peer Review Chart Rounds in    2011: A Survey of Academic Institutions in the United States. Int J    Radiat Oncol Biol Phys 2012; 84:590-5.-   [12] Moore K L, Brame R S, Low D a, Mutic S. Quantitative metrics    for assessing plan quality. Semin Radiat Oncol 2012; 22:62-9.-   [13] Ishikura S. Quality assurance of radiotherapy in cancer    treatment: toward improvement of patient safety and quality of care.    Jpn J Clin Oncol 2008; 38:723-9.-   [14] Marks L B, Jackson M, Xie L, Chang S X, Burkhardt K D, Mazur L,    et al. The challenge of maximizing safety in radiation oncology.    Pract Radiat Oncol 2011; 1:2-14.-   [15] Esch A Van, Bogaerts R, Kutcher G J, Van Esch A, Huyskens D.    Quality assurance in radiotherapy by identifying standards and    monitoring treatment preparation. Radiother Oncol 2000; 56:109-15.-   [16] Hendee W. Patient safety and the medical physicist. Med Phys    2011; 38:i-ii.-   [17] Hendee W R, Herman M G. Improving patient safety in radiation    oncology. Med Phys 2011; 38:78-82.-   [18] Ford E C, Terezakis S. How safe is safe? Risk in radiotherapy.    Int J Radiat Oncol Biol Phys 2010; 78:321-2.-   [19] Huq M S, Fraass B a, Dunscombe P B, Gibbons J P, Ibbott G S,    Medin P M, et al. A method for evaluating quality assurance needs in    radiation therapy. Int J Radiat Oncol Biol Phys 2008; 71:S170-3.-   [20] Furhang E E, Dolan J, Sillanpaa J K, Harrison L B. Automating    the initial physics chart checking process. J Appl Clin Med Phys    2009; 10:2855.-   [21] Yang D, Moore K L. Automated radiotherapy treatment plan    integrity verification. Med Phys 2012; 39:1542-51.-   [22] Zhao B, Joiner M C, Orton C G, Burmeister J. “SABER”: A new    software tool for radiotherapy treatment plan evaluation. Med Phys    2010; 37:5586-92.-   [23] Azmandian F, Kaeli D, Dy J G, Hutchinson E, Ancukiewicz M,    Niemierko A, et al. Towards the development of an error checker for    radiotherapy treatment plans: a preliminary study. Phys Med Biol    2007; 52:6511-24.-   [24] Nelms B E, Tomé W a, Robinson G, Wheeler J. Variations in the    contouring of organs at risk: test case from a patient with    oropharyngeal cancer. Int J Radiat Oncol Biol Phys 2012; 82:368-78.-   [25] Nelms B E, Robinson G, Markham J, Velasco K, Boyd S, Narayan S,    et al. Variation in external beam treatment plan quality: An    inter-institutional study of planners and planning systems. Pract    Radiat Oncol 2012; 2:296-305.-   [26] Moore K L, Kagadis G C, McNutt T R, Moiseenko V, Mutic S.    Vision 20/20: Automation and advanced computing in clinical    radiation oncology. Med Phys 2014; 41:010901.-   [27] Purdie T G, Dinniwell R E, Letourneau D, Hill C, Sharpe M B.    Automated planning of tangential breast intensity-modulated    radiotherapy using heuristic optimization. Int J Radiat Oncol Biol    Phys 2011; 81:575-83.-   [28] Good D, Lo J, Lee W R, Wu Q J, Yin F-F, Das S K. A    knowledge-based approach to improving and homogenizing intensity    modulated radiation therapy planning quality among treatment    centers: an example application to prostate cancer planning. Int J    Radiat Oncol Biol Phys 2013; 87:176-81.-   [29] Ghobadi K, Ghaffari H R, Aleman D M, Jaffray D A, Ruschin M.    Automated treatment planning for a dedicated multi-source    intracranial radiosurgery treatment unit using projected gradient    and grassfire algorithms. Med Phys 2012; 39:3134-41.-   [30] Voet P W J, Dirkx M L P, Breedveld S, Al-Mamgani A, Incrocci L,    Heijmen B J M. Fully automated volumetric modulated arc therapy plan    generation for prostate cancer patients. Int J Radiat Oncol Biol    Phys 2014; 88:1175-9.-   [31] Zhao X, Kong D, Jozsef G, Chang J, Wong E K, Formenti S C, et    al. Automated beam placement for breast radiotherapy using a support    vector machine based algorithm. Med Phys 2012; 39:2536-43.-   [32] Xhaferllari I, Wong E, Bzdusek K, Lock M, Chen J. Automated    IMRT planning with regional optimization using planning scripts. J    Appl Clin Med Phys 2013; 14.-   [33] Kazhdan M, Simari P, McNutt T, Wu B, Jacques R, Chuang M, et    al. A shape relationship descriptor for radiation therapy planning.    Lect Notes Comput Sci 2009; 5762:100-8.-   [34] Wu B, Ricchetti F, Sanguineti G, Kazhdan M, Simari P, Chuang M,    et al. Patient geometry-driven information retrieval for IMRT    treatment plan quality control. Med Phys 2009; 36:5497.-   [35] Webb S. Intensity-modulated radiation therapy. CRC Press; 2001.-   [36] Lee T, Hammad M, Chan T C Y, Craig T, Sharpe M B. Predicting    objective function weights from patient anatomy in prostate IMRT    treatment planning. Med Phys 2013; 40:121706.-   [37] Ehrgott M, Güler    , Hamacher H W, Shao L. Mathematical optimization in intensity    modulated radiation therapy. 4OR 2008; 6:199-262.-   [38] Marčelja S. Mathematical description of the responses of simple    cortical cells*. JOSA 1980:1297-300.-   [39] Fedkiw S O R. Level set methods and dynamic implicit surfaces.    Springer; 2003.-   [40] Goshtasby A A. 2-D and 3-D image registration: for medical,    remote sensing, and industrial applications. John Wiley & Sons;    2005.-   [41] Drucker H, Burges C J C, Kaufman L, Smola A, Vapnik V. Support    vector regression machines. Adv Neural Inf Process Syst 1997;    9:155-61.-   [42] Breiman L. Random Forests. Mach Learn 2001; 45:5-32.-   [43] Wu G, Wang Q, Zhang D, Shen D. Robust patch-based multi-atlas    labeling by joint sparsity regularization. MICCAI Work. STMI, 2012.-   [44] Gee J C, Reivich M, Bajcsy R. Elastically deforming 3D atlas to    match anatomical brain images. J Comput Assist Tomogr 1993;    17:225-36.-   [45] Zikic D, Glocker B, Criminisi A. Atlas encoding by randomized    forests for efficient label propagation. Med. Image Comput. Comput.    Interv. 2013, Springer; 2013, p. 66-73.-   [46] Bishop C M. Pattern Recognition and Machine Learning    (Information Science and Statistics). Springer-Verlag New York,    Inc.; 2006.-   [47] Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol P-A.    Stacked denoising autoencoders: Learning useful representations in a    deep network with a local denoising criterion. J Mach Learn Res    2010; 9999:3371-408.-   [48] McIntosh C, Svistoun I, Purdie T G. Groupwise conditional    random forests for automatic shape classification and contour    quality assessment in radiotherapy planning. IEEE Trans Med Imaging    2013; 32:1043-57.-   [49] Yang L, Jin R. Distance metric learning: A comprehensive    survey. Michigan State Universiy 2006; 2.-   [50] Osher S, Sethian J A. Fronts Propagating with Curvature    Dependent Speed: Algorithms Based on Hamilton-Jacobi Formulations. J    Comput Phys 1988; 79:12-49.-   [51] Criminisi A, Shotton J, Konukoglu E. Decision forests for    classification, regression, density estimation, manifold learning    and semi-supervised learning. Microsoft Res Cambridge, Tech . . .    2011.-   [52] Lafferty J D, McCallum A, Pereira F C N. Conditional Random    Fields: Probabilistic Models for Segmenting and Labeling Sequence    Data. ICML 2001:282-9.-   [53] Kolmogorov V, Rother C. Minimizing Nonsubmodular Functions with    Graph Cuts—A Review. IEEE Trans Pattern Anal Mach Intell 2007;    29:1274-9.-   [54] Boykov Y, Funka-Lea G. Graph Cuts and Efficient N-D Image    Segmentation. Int J Comput Vis 2006; 70:109-31.-   [55] Boykov Y Y, Jolly M-P. Interactive graph cuts for optimal    boundary & region segmentation of objects in N-D images. Comput    Vision, 2001 ICCV 2001 Proceedings Eighth IEEE Int Conf 2001;    1:105-12.-   [56] Papadimitriou C H, Steiglitz K. Combinatorial optimization:    algorithms and complexity. Dover Publications Inc., Mineola, N Y;    1998.-   [57] Pati Y C, Rezaiifar R, Krishnaprasad P S. Orthogonal matching    pursuit: Recursive function approximation with applications to    wavelet decomposition. Signals, Syst. Comput. 1993. 1993 Conf. Rec.    Twenty-Seventh Asilomar Conf., IEEE; 1993, p. 40-4.-   [58] Olshausen B A, Field D J. Sparse coding with an overcomplete    basis set: a strategy employed by V1? Vision Res 1997; 37:3311-25.-   [59] Wu T T, Lange K. Coordinate descent algorithms for lasso    penalized regression. Ann Appl Stat 2008:224-44.-   [60] Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A. Supervised    dictionary learning 2008.-   [61] Blumensath T, Davies M E. On the difference between orthogonal    matching pursuit and orthogonal least squares 2007.-   [62] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning    applied to document recognition. Proc IEEE 1998; 86:2278-324.-   [63] Hinton G E, Osindero S, Teh Y-W. A fast learning algorithm for    deep belief nets. Neural Comput 2006; 18:1527-54.-   [64] Salakhutdinov R, Hinton G E. Deep boltzmann machines. Int.    Conf. Artif. Intell. Stat., 2009, p. 448-55.-   [65] Lee H, Grosse R, Ranganath R, Ng A Y. Convolutional deep belief    networks for scalable unsupervised learning of hierarchical    representations. Proc. 26th Annu. Int. Conf. Mach. Learn., ACM;    2009, p. 609-16.-   [66] Coates A, Ng A Y. The importance of encoding versus training    with sparse coding and vector quantization. Proc. 28th Int. Conf.    Mach. Learn., 2011, p. 921-8.-   [67] Qazi A A, Pekar V, Kim J, Xie J, Breen S L, Jaffray D A.    Auto-segmentation of normal and target structures in head and neck C    T images: a feature-driven model-based approach. Med Phys 2011;    38:6160-70.-   [68] Pekar V, McNutt T R, Kaus M R. Automated model-based organ    delineation for radiotherapy planning in prostatic region. Int J    Radiat Oncol Biol Phys 2004; 60:973-80.-   [69] Chang C-C, Lin C-J. LIBSVM: A library for support vector    machines. ACM Trans Intell Syst Technol 2011; 2:27:1-27:27.-   [70] Mohri C C M. Confidence intervals for the area under the ROC    curve. Adv Neural Inf Process Syst 17 Proc 2004 Conf 2005; 17:305.-   [71] Borg I, Groenen P J F. Modern multidimensional scaling: Theory    and applications. Springer; 2005.

The invention claimed is:
 1. A method for generating a proposedtreatment plan for radiation therapy, the method comprising: obtaining aset of patient data for a patient including a set of image data for atleast one treatment site; determining a treatment plan class, from aplurality of predefined treatment plan classes, each predefinedtreatment plan class defining one or more treatment plan featuresrelevant to treatment of a respective treatment site; calculating aproposed dose map by determining a dosage over a volume depicted in theset of image data according to a first set of rules defining expectedrelationships between an applied dosage, the determined treatment planclass, and at least one feature of the set of image data; determiningone or more treatment plan parameters for achieving the proposed dosemap according to a second set of rules defining expected relationshipsbetween the applied dosage and treatment plans; generating as output theproposed treatment plan including the one or more determined treatmentplan parameters; and displaying a visualization of the proposed dose mapon a display device, wherein the visualization further comprisesproviding, with the visualization, a user interface for receiving userinput, wherein the first set of rules includes at least one of a rulegenerated by computer learning based on historical data, a mathematicalfunction relating dose values of the proposed dose map to the at leastone feature of the set of image data, or a general rule governing dosemaps entered manually.
 2. The method of claim 1, wherein determining thetreatment plan class comprises: determining the treatment plan classbased on the patient data; and using an automated classificationalgorithm, wherein the treatment plan class is determined based on oneor more of: a patient's electronic medical record; similarity offeatures of the set of image data with image features of a giventreatment plan class; presence of one or more defined regions ofinterest (ROIs) corresponding to one or more ROIs of a given treatmentplan class; manually inputted data; and metadata provided with the imagedata.
 3. The method of claim 1, further comprising: obtaining a set ofregion of interest (ROI) data delineating at least one ROI in the set ofimage data; wherein calculating the proposed dose map includesdetermining a dosage over the at least one ROI according to the firstset of rules, the at least one feature of the set of image datacomprising the at least one ROI, and further comprising: automaticallycharacterizing the at least one ROI according to one or more predefinedfeatures to determine at least one ROI characterization; wherein thefirst set of rules comprises at least one rule defining expectedrelationships between the applied dosage and the at least one ROIcharacterization.
 4. The method of claim 3, wherein calculating theproposed dose map further includes determining a dosage over a volume ofthe image data according to the first set of rules, using a non-ROIfeature of the set of image data.
 5. The method of claim 4, wherein thefirst set of rules includes at least one rule defining an expectedrelationship between the applied dosage and the at least one ROI, and atleast one rule defining an expected relationship between the applieddosage and the non-ROI feature of the image data.
 6. The method of claim5 wherein the second set of rules includes one or more of: historicaltreatment parameters for a given planned dosage; a mathematicalfunction; and a general rule governing treatment parameters irrespectiveof the proposed dose map.
 7. The method of claim 1, wherein the proposeddose map defines voxel-by-voxel a range of possible doses, furthercomprising, prior to determining the one or more treatment planparameters, adjusting the proposed dose map to define specificvoxel-by-voxel doses within the range of possible doses according to athird set of rules defining expected relationships between the applieddosage and the determined treatment plan class.
 8. The method of claim1, wherein the visualization comprises a voxel-by-voxel mapping ofproposed dosages superimposed on the set of image data and avoxel-by-voxel indication of a confidence measure for each voxel of theproposed dose map, wherein the user input received by the user interfacemodifies at least one of: proposed dosage for at least one voxel of theproposed dose map, and one or more features of the image data; andrecalculating the proposed dose map in accordance with an inputtedmodification.
 9. The method of claim 1, wherein the first set of rulesis the rule generated by computer learning based on historical data. 10.The method of claim 1, wherein the first set of rules is themathematical function relating dose values of the proposed dose map tothe at least one feature of the set of image data.
 11. The method ofclaim 1, wherein the first set of rules is the general rule governingdose maps entered manually.